PaperlayerView exact revision on arXiv

Source-linked AI summary

CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding

Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho

arXiv:2509.23379v2cs.CLcs.AIcs.CV

TL;DR

Radiology MLLMs can produce clinically unsupported descriptions, and prompt-induced hallucinations remain prevalent amid over-sensitivity to clinical sections. CCD uses task-specific radiology expert signals in a training-free, retrieval-free inference framework; experiments report improved RRG linguistic quality and clinical fidelity as well as VQA accuracy.

  • Problem

    Radiology MLLMs can generate image-unsupported or diagnostically misaligned content, and prompt-induced hallucinations remain prevalent despite fine-grained inputs.

    Source: §1 · pp. 1–2

  • Method

    CCD uses task-specific expert-derived clinical labels and probabilities in a two-stage inference-time decoding framework without retraining.

    Source: §1 · p. 2

  • Results

    Across three datasets and multiple models, CCD improves linguistic quality and clinical fidelity in RRG and accuracy on VQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study presents expert models as a complementary component for improving clinical accuracy in safety-critical medical imaging tasks.

    Source: §6 · p. 9, G EXTENDED DISCUSSION ON LIMITATIONS · p. 27

    Clinical applications impose more stringent safety, reliability, and interpretability requirements than general-purpose AI tasks.

    Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have recently achieved remarkable progress in radiology by integrating visual perception with natural language understanding. However, they often generate clinically unsupported descriptions, known as medical hallucinations, which pose serious risks in medical applications that demand accuracy and image-grounded outputs. Through empirical analysis, we find that prompt-induced hallucinations remain prevalent in radiology MLLMs, largely due to over-sensitivity to clinical sections. To address this, we introduce Clinical Contrastive Decoding (CCD), a training-free and retrieval-free inference framework that integrates structured clinical signals from task-specific radiology expert models. CCD introduces a dual-stage contrastive mechanism to refine token-level logits during generation, thereby enhancing clinical fidelity without modifying the base MLLM. Experiments on three datasets and multiple models demonstrate that CCD consistently improves overall performance on radiology report generation (RRG). On the MIMIC-CXR dataset, it yields up to a 17% improvement in RadGraph-F1 when applied to state-of-the-art RRG models. Our approach provides a lightweight and generalisable solution for mitigating medical hallucinations, effectively bridging expert models and MLLMs in radiology.

1 INTRODUCTION

Radiology MLLMs can produce clinically plausible but image-unsupported content, and misleading clinical sections can trigger such hallucinations. The paper introduces CCD, which uses expert-derived clinical signals to guide generation, and reports gains across radiology report generation and visual question answering.

1 INTRODUCTION

  • Radiology requires factual accuracy and clinical reliability, while medical hallucinations include image-unsupported or diagnostically misaligned outputs.The introduction notes that such errors can adversely affect diagnosis and patient treatment.Source: §1 · p. 1
  • CCD uses task-specific expert models to provide structured clinical labels and confidence-based guidance during inference, without further alignment or retraining.Predicted labels are injected as descriptive prompts, while probabilities perturb decoding.Source: §1 · p. 2
  • Prompt-induced medical hallucinations remain prevalent in radiology MLLMs and often stem from over-sensitivity to clinical sections.The paper frames these errors as a reason to seek inference-time solutions beyond dataset-level interventions.Source: §1 · p. 2
  • Experiments across three datasets and multiple models show improved linguistic quality and clinical fidelity in RRG, as well as improved VQA accuracy.Source: §1 · p. 2

2 RELATED WORK

Prior radiology hallucination mitigation has emphasized training-time changes and retrieval, while general inference-time contrastive methods have not transferred well to complex radiology tasks. The paper positions CCD among lightweight inference-time approaches that use domain-specific clinical signals.

Radiology Multimodal Large Language Models

  • Radiology MLLMs have advanced, but medical hallucination remains a limitation to their clinical reliability.Source: §2 · p. 3

Medical Hallucination in Multimodal Large Language Models

  • Prior medical-hallucination mitigation has focused mainly on training-time interventions or retrieval, while voting-based inference methods for VQA do not generalise well to complex RRG.Source: §2 · p. 3

Radiology Report Generation

  • RRG approaches use data sanitization, retrieval, structured clinical sections, or temporal modeling, but may require costly curation or retrieval infrastructure.Source: §2 · p. 3

Contrastive Decoding Strategies

  • General-domain contrastive decoding methods use visual or textual perturbations and other inference cues, but face challenges in radiology, including grayscale images and limited domain-specific data.Radiology MLLMs are also often trained for single tasks, limiting generalisation of training-free strategies.Source: §2 · p. 3

3 MEDICAL HALLUCINATION IN RADIOLOGY MLLMS

The study probes prompt-induced hallucinations in chest X-ray report generation by varying clinical sections in prompts to LLaVA-Med v1.5. Appended sections can make reports more fluent while degrading clinical metrics, and comparison text that cannot be verified from the image is linked to hallucination risk.

Setup for Medical Hallucinations

  • The analysis uses MIMIC-CXR chest X-rays and varies indication, technique, comparison, or history sections appended to LLaVA-Med v1.5 prompts.The study uses greedy decoding and evaluates factual inconsistency handling alongside report quality.Source: §3 · p. 4

Hallucination Drivers: Clinical Context Sensitivity

  • Clinical terminology in history or technique sections can slightly improve lexical scores by making generated text resemble radiology-report style.Source: §3 · pp. 4–5
  • ↓8.12 BERTScore with comparison sections, whose references to prior exams or temporal change cannot be observed in the current frontal image.The text-image mismatch introduces unverifiable context and increases the likelihood of hallucinated content.Source: §3 · p. 5
  • ↑6.90 Cardiomegaly CheXbert-F1 with indication sections, despite a general decline in clinical report quality across appended sections.Pleural Effusion and Atelectasis tend to decrease; the authors associate this pattern with overemphasis on text at the expense of subtle visual evidence.Source: §3 · p. 5
  • Because original report sections can provide misleading signals, the authors motivate selecting clinically relevant, context-appropriate information during inference.CCD is proposed to use expert models for accurate, image-grounded clinical information.Source: §3 · p. 5

4 CLINICAL CONTRASTIVE DECODING

Clinical Contrastive Decoding adjusts generation with expert-derived clinical signals in two inference stages. The stages target false negatives and false positives to improve clinical reliability.

4 CLINICAL CONTRASTIVE DECODING

  • CCD combines symptom-grounded and expert-informed decoding to adjust MLLM token logits during inference and mitigate false negatives and false positives.The framework hierarchically calibrates output logits toward ground-truth clinical labels.Source: §4 · p. 5, Figure 2 · p. 5, §4.2 · p. 7

4.1 SYMPTOM-GROUNDED CONTRASTIVE DECODING (SCD)

  • SCD uses expert-derived symptom labels to condition a contrastive branch, guiding the MLLM toward relevant findings and reducing false negatives.The method focuses on 14 CheXpert pathology labels; selected labels form an anchor prompt for the image-conditioned generation path.Source: §4.1 · p. 6
  • SCD contrasts anchor-conditioned and original outputs using normalized log-probabilities to balance clinical guidance against excessive reliance on context.The contrastive branch is intended to suppress false negatives, while log-softmax helps manage scale and shift differences between output distributions.Source: §4.1 · p. 6

4.2 EXPERT-INFORMED CONTRASTIVE DECODING (ECD)

  • ECD transforms expert symptom probabilities into token-aligned logit biases and clips their magnitude to limit over-correction.The bias is incorporated into the first-stage signal, with clipping bounds set using γ values of 2, 5, or 10.Source: §4.2 · pp. 6–7
  • ECD applies expert-informed constraints to suppress clinically unjustified symptoms, then interpolates its output with adjusted SCD logits for final decoding.The interpolation balances internal contrastive and expert-informed logits while maintaining linguistic fluency.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments assess CCD across radiology report generation and visual question answering, using radiology MLLMs, expert guidance, and comparisons with training-free decoding baselines. Results and ablations examine clinical and lexical performance, stage contributions, expert settings, and guidance strength.

5.1 EXPERIMENTAL SETTINGS

  • Experiments span three report-generation datasets and Medical-CXR-VQA, comparing CCD with greedy decoding and recent training-free mitigation methods.MAIRA-2 is used for report generation and LLaVA-Med for VQA; the default expert is a TorchXRayVision pathology classifier.Source: §5 · p. 7, §5.1 · pp. 7–8
  • CCD uses fixed cross-task settings of α = 0.5, β = 0.5, and γ = 10, alongside standard metric evaluation for report generation and VQA.VQA scores are micro-averaged Recall and F1 based on reference-label mentions in generated text.Source: §5.1 · pp. 7–8

5.2 EXPERIMENTAL RESTULTS

  • 67% higher CheXbert5 F1 and 17% higher RadGraph-F1 on MIMIC-CXR are reported for CCD over the baseline in report generation.Table 2 evaluates radiology report generation; results on IU-Xray and CheXpert Plus are reported only for CCD.Source: Table 2 · p. 8, §5.2 · p. 8
  • CCD improves most Medical-CXR-VQA categories, with slight drops for Location and Type and competitive overall performance despite those cases.The authors attribute the drops mainly to morphological findings not well captured by the 14-category expert model.Source: Table 3 · p. 8, §5.2 · pp. 8–9

5.3 ABLATION STUDIES

  • Removing SCD notably reduces CheXbert5,14 F1 and symptom coverage; removing ECD causes a smaller clinical-metric drop but slightly improves some lexical scores.Removing both stages causes the largest overall degradation, supporting their complementary roles.Source: §5.3 · p. 9
  • Restricting expert output to the top five symptoms causes a larger CheXbert14 F1 drop of 0.27, while replacing the default expert with MedSigLIP improves both metric types.Top-five output slightly improves lexical and some clinical metrics, but the authors emphasize broad label coverage; the MedSigLIP comparison supports robustness across expert settings.Source: §5.3 · p. 9
  • RadGraph-F1 is best when symptom-guidance α and expert-confidence β are both 0.5.The study varies both guidance weights while holding other settings at default.Source: §5.3 · p. 9

6 CONCLUSION

The conclusion presents CCD as an inference-time framework that uses expert signals and dual-stage logit interventions to improve clinical consistency in radiology generation. Experiments across models, datasets, and metrics support its effectiveness in radiology tasks.

6 CONCLUSION

  • CCD is a training-free, retrieval-free inference framework that uses a task-specific expert model and dual-stage interventions on MLLM latent logits.The conclusion describes the approach as requiring neither retraining nor data augmentation.Source: §6 · p. 9
  • Experiments across diverse models, datasets, and metrics validate CCD's effectiveness in radiology tasks, including RRG and VQA.The conclusion reports improved clinical consistency in RRG and contributions to VQA performance.Source: §6 · p. 9
  • The conclusion frames expert-model guidance as a practical path toward more trustworthy, clinically aligned systems, while describing CCD as a modest step toward physician-level reliability.Detailed results and the plausibility-constraint ablation are noted in Appendix E.Source: §6 · p. 9

ETHICS STATEMENT

The paper frames CCD as an inference-time approach to reducing clinically important prompt-induced hallucinations, while setting clear limits on its scope. It reports use of publicly available, de-identified data and emphasizes clinician oversight rather than replacement.

ETHICS STATEMENT

  • The study uses publicly available, de-identified data and says figures blur chest X-rays to preserve privacy and minimize visual discomfort.The authors state that they follow the datasets' ethical guidelines and usage policies.Source: ETHICS STATEMENT · p. 10
  • The paper positions medical AI as assistance for licensed clinicians, requiring qualified radiologist oversight and caution against excessive reliance on automated outputs.It also flags risks to human involvement and existing healthcare disparities.Source: ETHICS STATEMENT · p. 10

A.1 RESEARCH AIMS

  • CCD targets clinically important prompt-induced hallucinations without changing model parameters or requiring additional training, but does not aim to eliminate all medical hallucinations.The work focuses on test-time decoding and excludes comparisons with architecture changes, additional training, or retrieval augmentation.Source: §A.1 · p. 18

A.2 RESEARCH SCOPE

  • The experiments are limited to pretrained radiology MLLMs on frontal AP and PA chest X-rays for RRG and VQA.The scope excludes other imaging modalities, clinical-note or EHR signals, general-domain MLLMs, and post-processing approaches.Source: §A.2 · p. 18

B.1 DATASETS DESCRIPTION

The study evaluates report generation on three chest X-ray datasets and VQA on Medical-CXR-VQA, with task-specific lexical, clinical, and label-inclusion metrics. The datasets and metric procedures define the scope and interpretation of the evaluations.

B.1 DATASETS DESCRIPTION

  • MIMIC-CXR evaluation uses frontal AP or PA images and a test split of 2,461 studies with non-empty Findings sections; IU-Xray evaluation uses 3,307 frontal-view cases with non-empty Findings.MIMIC-CXR is a large-scale dataset of 377,110 radiographs from 227,835 studies; IU-Xray contains 7,470 images and 3,955 reports.Source: §B.1 · pp. 18–19
  • CheXpert Plus evaluation uses 72 frontal-view validation samples because its official test split is unavailable; Medical-CXR-VQA uses 78,124 official test image-question pairs.Medical-CXR-VQA covers six predefined question types and is derived from MIMIC-CXR.Source: §B.1 · p. 19

B.2 EVALUATION METRICS

  • Report generation is assessed with lexical overlap metrics and radiology-specific clinical metrics, including RadGraph-F1, Temporal-F1, RaTeScore, RadEval-BERT, and CheXbert-F1.CheXbert-F1 is reported for both 14 classes and a five-class version; metrics use the RadEval toolkit's default configurations.Source: §B.2 · p. 19
  • VQA uses micro-averaged Recall and F1 based on whether reference labels appear in generated text, counting missing labels as false negatives and not penalizing false positives.Micro-averaging weights frequent conditions appropriately in sparse multi-label settings, according to the authors.Source: §B.2 · p. 19

C EXPERIMENTAL DETAILS

Experiments use pretrained radiology MLLMs and expert classifiers with training-free decoding, and compare CCD with other decoding strategies. CCD improves lexical and clinical metrics and is reported as more effective than general-domain strategies for radiology-specific generation.

C EXPERIMENTAL DETAILS

  • CCD requires no additional training and adds approximately 1.45× inference-time overhead over standard greedy decoding.The reported runtime overhead may vary with hardware, particularly GPU FLOPS.Source: C EXPERIMENTAL DETAILS · p. 20

C.1 BACKBONE MODELS

  • The evaluated backbones span grounded-report MAIRA-2, longitudinal Libra, report-tuned LLaVA-Rad, and biomedical-VQA LLaVA-Med.The models use different vision-language components and training setups.Source: §C.1 · p. 20

C.2 METHOD CONFIGURATION

  • All comparison methods use their published hyperparameter recommendations without additional tuning beyond those settings.MAIRA-2 uses default LLaVA-type settings for training-free decoding methods.Source: §C.2 · p. 20

C.3 EXPERT MODEL SETTING

  • The expert guidance comes from DenseNet pathology probabilities or MedSigLIP zero-shot symptom probabilities.DenseNet predicts 14 predefined pathologies; MedSigLIP compares positive and negative symptom prompts.Source: §C.3 · p. 21

D.1 COMPARISON OF DECODING STRATEGIES ON RADIOLOGY REPORT GENERATION

  • CCD consistently improves lexical and clinical metrics and is more effective than general-domain decoding strategies for radiology report generation.Table 5 compares decoding methods using MAIRA-2 as the baseline; results on IU-Xray and CheXpert Plus are reported only for CCD.Source: §D.1 · p. 22, Table 5 · p. 22

D.2 COMPARISON OF BACKBONE MLLMS FOR RADIOLOGY REPORT GENERATION

Across multiple backbones and datasets, CCD improves overall radiology report generation performance, particularly on clinical metrics. The results also show a possible trade-off between clinical consistency and lexical quality.

D.2 COMPARISON OF BACKBONE MLLMS FOR RADIOLOGY REPORT GENERATION

  • The evaluation covers Libra, LLaVA-Rad, and LLaVA-Med on MIMIC-CXR, IU-Xray, and CheXpert Plus, using default CCD settings without tuning.Those fixed settings may under-optimize performance for some backbones.Source: §D.2 · pp. 22–23
  • Across backbones, CCD consistently improves overall performance, particularly on clinical metrics.The comparison is against greedy decoding without clinical-section input.Source: Table 6 · p. 23, §D.2 · p. 23
  • LLaVA-Med gains 1.64× in CheXbert5 F1, while showing slight decreases in lexical metrics.Source: §D.2 · p. 23

E.1 IMPACT OF VARYING CONTROL STRENGTH IN CLINICAL CONTRASTIVE DECODING

Ablations vary α, β, and γ on MAIRA-2 using MIMIC-CXR to examine how guidance strength affects report generation. The results show a balance between clinical and lexical metrics, while randomized expert signals cause no significant degradation.

E.1 IMPACT OF VARYING CONTROL STRENGTH IN CLINICAL CONTRASTIVE DECODING

  • As α rises from 0 to 1, lexical and CheXbert-based scores improve, but RadGraph-F1 and RaTeScore begin to degrade above 0.5.The authors use α = 0.5 by default because α = 1 can produce overly verbose descriptions.Source: Table 7 · p. 24, §E.1 · p. 24
  • As β increases, CheXbert-based clinical scores improve while lexical scores decrease; RadGraph-F1, Temporal-F1, and RaTeScore decline above 0.5.Strong probabilistic constraints can overfocus generation on symptom descriptions and reduce fluency; β = 0.5 is the default.Source: Table 8 · p. 24, §E.1 · p. 24
  • The best balance across lexical and clinical metrics occurs at γ = 10, the default associated with a clinically meaningful threshold for severe diagnostic evidence.Some metrics, including RadEval-BERT and F1, peak at lower constraint strengths.Source: §E.1 · pp. 24–25, Table 9 · p. 25

E.2 ROBUSTNESS TEST OF CLINICAL CONTRASTIVE DECODING WITH RANDOM PRIOR

  • Replacing expert signals with random values causes no significant degradation across lexical or clinical metrics.The test uses MAIRA-2 on MIMIC-CXR with default CCD hyperparameters.Source: Table 10 · p. 25, §E.2 · p. 25

F BALANCING ACCURACY AND AMBIGUITY

With fixed hyperparameters, CCD slightly lowers accuracy in the Location and Type VQA categories, where some broadly phrased answers are clinically reasonable but judged incorrect. Lowering α can make answers more specific, illustrating a trade-off between cautiousness and specificity.

F BALANCING ACCURACY AND AMBIGUITY

  • Fixed hyperparameters across models and question types accompany marginal accuracy decreases in Location and Type, despite overall VQA improvement.The authors avoid type-specific tuning to prioritize generality and plug-and-play use.Source: F BALANCING ACCURACY AND AMBIGUITY · p. 25
  • Some answers judged incorrect in Location and Type are ambiguous but clinically reasonable, reflecting borderline judgments under strict evaluation.Figure 4 presents examples from both categories; blue highlights indicate vague or under-specified outputs.Source: Figure 4 · p. 26, F BALANCING ACCURACY AND AMBIGUITY · p. 26
  • The initial anchor stage favors common CheXpert-related symptoms, sometimes under-representing more specific concepts and producing conservative answers.The passage characterizes these as possible “dummy” false negatives.Source: F BALANCING ACCURACY AND AMBIGUITY · p. 26
  • Reducing α from 0.5 to 0.3 softens first-stage guidance and improves answer specificity in Location and Type.The authors note that question categories may differ in sensitivity to CCD control parameters.Source: F BALANCING ACCURACY AND AMBIGUITY · p. 26
  • Cautious responses may sacrifice specificity, while assertive responses can introduce misleading or incorrect information.Source: F BALANCING ACCURACY AND AMBIGUITY · p. 26

G EXTENDED DISCUSSION ON LIMITATIONS

The authors flag limits in dataset generalisability, evaluation, model access, and robustness under real-world distribution shifts. They conclude that CCD offers a way to combine MLLMs with specialised expert models to improve clinical accuracy.

G EXTENDED DISCUSSION ON LIMITATIONS

  • The authors caution that clinical applications impose stricter safety, reliability, and interpretability requirements than general-purpose AI tasks.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27
  • Both evaluation datasets come from one institution, potentially limiting generalisability to settings with different patient populations or workflows.The authors cite possible institution-specific biases in demographics, imaging protocols, and reporting practices.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27
  • All evaluations use automatic metrics as relative references to ground truth, and the authors recommend reader studies or licensed-radiologist review for clinical plausibility and safety.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27
  • Reliance on publicly available models and third-party APIs limits controlled experiments and scaling analysis, while targeted logit modification requires direct access to latent logits.Only the 7B MAIRA-2 variant is open-sourced in the described setting.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27
  • Robustness remains to be evaluated under distribution shifts involving lower-quality portable X-rays and heterogeneous imaging equipment.The authors note that many radiology MLLMs and expert models are trained on well-curated data with standardised image quality and controlled acquisition conditions.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27
  • The authors frame CCD as one way to combine generative foundation models with specialised expert models for improving clinical accuracy in safety-critical medical imaging.Source: G EXTENDED DISCUSSION ON LIMITATIONS · p. 27

Related papers

Loading 2509.23379v2…