PaperlayerView exact revision on arXiv

Source-linked AI summary

RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models

Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, Huaxiu Yao

arXiv:2407.05131v2cs.LGcs.AIcs.CLcs.CVcs.CY

TL;DR

Med-LVLMs can generate factually inaccurate responses, and RAG can either provide insufficient context or lead models to over-rely on retrieved information. RULE combines calibrated retrieval-count risk control with preference fine-tuning, and reports a 47.4% average accuracy improvement over MedVInT across datasets.

  • Problem

    Med-LVLMs can depart from medical facts, while RAG may provide too little context, introduce inaccurate references, or overturn an otherwise correct answer through over-reliance.

    Source: §1 · pp. 1–2

  • Method

    RULE calibrates the number of retrieved contexts to control factuality risk and preference-fine-tunes Med-LVLMs using retrieval-induced errors as dispreferred samples and ground-truth responses as preferred samples.

    Source: §1 · p. 2

  • Results

    RULE achieves an average accuracy improvement of 47.4% over MedVInT and state-of-the-art performance across all datasets.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Experiments on three medical imaging analysis datasets demonstrate RULE's effectiveness in addressing factuality challenges in medical RAG.

    Source: §6 · p. 9

    The work focuses on factual accuracy; safety, fairness, robustness, and privacy in clinical deployment remain for future research.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The recent emergence of Medical Large Vision Language Models (Med-LVLMs) has enhanced medical diagnosis. However, current Med-LVLMs frequently encounter factual issues, often generating responses that do not align with established medical facts. Retrieval-Augmented Generation (RAG), which utilizes external knowledge, can improve the factual accuracy of these models but introduces two major challenges. First, limited retrieved contexts might not cover all necessary information, while excessive retrieval can introduce irrelevant and inaccurate references, interfering with the model's generation. Second, in cases where the model originally responds correctly, applying RAG can lead to an over-reliance on retrieved contexts, resulting in incorrect answers. To address these issues, we propose RULE, which consists of two components. First, we introduce a provably effective strategy for controlling factuality risk through the calibrated selection of the number of retrieved contexts. Second, based on samples where over-reliance on retrieved contexts led to errors, we curate a preference dataset to fine-tune the model, balancing its dependence on inherent knowledge and retrieved contexts for generation. We demonstrate the effectiveness of RULE on medical VQA and report generation tasks across three datasets, achieving an average improvement of 47.4% in factual accuracy. We publicly release our benchmark and code in https://github.com/richard-peng-xia/RULE.

1 Introduction

Med-LVLMs can produce responses that depart from medical facts, while RAG can both miss needed information and introduce misleading contexts. RULE combines calibrated retrieval-count selection with preference fine-tuning, and reports improved factual accuracy across medical benchmarks.

1 Introduction

  • Med-LVLMs remain prone to responses that deviate from medical facts, potentially risking clinical decisions and adverse outcomes.The passage frames factual alignment as important for medical applications.Source: §1 · p. 1
  • RAG can lack needed context when retrieval is small and introduce low-relevance or inaccurate references when retrieval is large.The figure also illustrates that retrieved contexts can overturn an otherwise correct response through model over-reliance.Source: Figure 1 · p. 1, §1 · p. 2
  • RULE controls factuality risk by calibrating the number of retrieved contexts and balances internal knowledge with retrieval through preference fine-tuning.Its preference data treats retrieval-induced errors as dispreferred and ground-truth responses as preferred.Source: §1 · p. 2
  • 14.46% improvement over the best prior methods for mitigating hallucination is reported across three medical VQA and report-generation benchmarks.The introduction also reports component effectiveness and compatibility.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe Med-LVLM inputs and outputs alongside preference optimization. They introduce preference pairs and the DPO classification-loss formulation, while Figure 2 sketches RULE's retrieval-risk control and preference-tuning components.

2 Preliminaries

  • Med-LVLMs combine LLMs with medical visual modules, taking medical images and clinical queries as input and autoregressively predicting text.The text output is denoted y.Source: §2 · p. 2
  • Preference data pairs an input prompt with preferred and dispreferred responses, and DPO formulates optimization over that data as a classification loss.The preference-pair probability is expressed using a sigmoid of the reward difference.Source: §2 · p. 2
  • The reference policy is the LLM fine-tuned through supervised learning; Figure 2 depicts RULE's calibrated retrieval-count risk control and preference tuning.The tuning data targets errors caused by excessive reliance on retrieved contexts.Source: §2 · p. 2, Figure 2 · p. 3

3 Methodology

RULE combines multimodal reference retrieval with calibrated selection of retrieval count and preference tuning intended to balance model knowledge against retrieved contexts. Its risk-control procedure evaluates candidate retrieval counts against factuality-risk tolerance, while KBPT uses retrieval-induced errors as preference data.

3 Methodology

  • RULE brings together reference retrieval, calibrated selection of retrieved-context count, and preference optimization to balance the model’s own knowledge against retrieved contexts.The methodology presents these as three modules working together.Source: §3 · p. 3

3.1 Context Retrieval for Reference

  • RULE retrieves reports similar to the target image as references; vision and text encoders are contrastively fine-tuned to match same-example image-report embeddings and separate different examples.The retrieved reports guide generation for the target medical image.Source: §3.1 · pp. 3–4

3.2 Factuality Risk Control Through Calibrated Retrieved Context Selection

  • For each candidate k, RULE estimates factuality risk as 1−ACC, combines two test probabilities, and uses family-wise error control to retain values meeting tolerance, with probability at least 1−δ.The two probabilities use different approaches; the procedure takes their minimum before applying an error-rate-controlling selection method.Source: §3.2 · p. 4

3.3 Knowledge Balanced Preference Tuning

  • Because about half of retrieval-augmented errors involve over-reliance, KBPT pairs originally correct answers later made incorrect by retrieval as dispreferred against ground truth, then fine-tunes with direct preference optimization.The preference samples come from a separate set not used to fine-tune the retriever.Source: §3.3 · pp. 4–5

4 Experiment

The experiments evaluate RULE across medical VQA and report generation, comparing it with mitigation methods and open-source Med-LVLMs. They report improved factual accuracy and examine the contributions of its components, compatibility, and case examples.

4.1 Experimental Setups

  • RULE is evaluated on MIMIC-CXR, IU-Xray, and Harvard-FairVLMed, using accuracy for VQA and BLEU, ROUGE-L, and METEOR for report generation.The study compares against decoding methods and open-source Med-LVLMs.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Results

  • In comparisons with various hallucination-reduction methods, RULE demonstrates the best overall performance, with a 47.4% average accuracy improvement on two tasks.Source: §4.2 · p. 6
  • 47.4% average accuracy improvement over MedVInT is reported, with RULE achieving state-of-the-art performance across all datasets.The comparison covers open-source Med-LVLMs and includes radiology and ophthalmology results.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 How Does RULE Improve the Performance?

  • The ablation analysis reports that retrieval alone slightly improves factual accuracy on two datasets but decreases it on MIMIC-CXR; adding factuality risk control yields stable gains, and KBPT further improves reliability.Combining factuality risk control and KBPT achieves optimal performance.Source: §4.3 · p. 7, Table 5 · p. 7
  • After KBPT, error and over-reliance rates decrease by averages of 42.9% and 47.3%, while attention shifts away from retrieved contexts and toward questions.Figure 3 visualizes the error and over-reliance measures alongside text and image attention maps.Source: §4.3 · p. 7, Figure 3 · p. 7
  • KBPT trained with VQA preference data performs best across all three datasets; captioning data may improve fact description but does not teach balancing retrieved content with model knowledge.The preference samples are cases where the model erred through over-reliance on retrieved contexts.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 Compatibility Analysis

  • On LLaVA-Med-1.0, RULE increases accuracy by an average of 16.7%, supporting KBPT compatibility across different models and datasets.The analysis applies KBPT to LLaVA-Med-1.0 and reports gains on three datasets.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 Case Study

  • Two radiology and ophthalmology cases illustrate that RULE can address factual errors persisting after RAG and errors arising when retrieved contexts override an initially correct answer.The examples contrast these cases with LLaVA-Med’s initial responses.Source: §4.5 · p. 8, Figure 5 · p. 8

5 Related Work

Related work describes factual errors in Med-LVLMs and the use of retrieval-augmented generation to improve factual responses. It identifies retrieval quantity and over-reliance on references as unresolved challenges in medical multimodal applications.

Factuality in Med-LVLMs

  • Med-LVLMs can produce responses that conflict with visual medical information, creating potential for misdiagnosis or missed diagnosis.Benchmarks evaluate factuality in VQA and report generation, while improving factual accuracy remains underexplored.Source: §5 · p. 8

Retrieval Augmented Generation

  • RAG has been applied to medical VQA and report generation, but the number of retrieved contexts and model over-reliance on references remain concerns in Med-LVLMs.The passage notes that these factors may affect performance or degrade it.Source: §5 · p. 9

6 Conclusion

RULE combines calibrated retrieval-context selection with preference optimization to improve Med-LVLM factuality. Experiments on three medical imaging analysis datasets demonstrate its effectiveness.

6 Conclusion

  • RULE calibrates the number of retrieved contexts to control factuality risk.It also uses preference optimization to address errors from excessive dependence on retrieved information.Source: §6 · p. 9
  • Preference optimization aims to balance Med-LVLMs’ intrinsic knowledge with retrieved information.Source: §6 · p. 9
  • Experiments on three medical imaging analysis datasets demonstrate RULE’s effectiveness.Source: §6 · p. 9

Limitations

The paper focuses on factual accuracy in Med-LVLMs and identifies other clinical deployment issues as future research directions.

Limitations

  • The study focuses on factual accuracy; safety, fairness, robustness, and privacy are future research topics.Source: Limitations · p. 9

A.1 Data statistics

The appendix describes the data statistics and construction of the preference-tuning dataset, along with steps used to prepare and check question-answer pairs.

A.1 Data statistics

  • Training statistics distinguish retriever image-caption pairs from KBPT question-answer pairs.Tables 7 and 8 present training- and test-set statistics, respectively.Source: §A.1 · p. 12, Table 7 · p. 12, Table 8 · p. 12
  • “All” includes samples whose originally correct answers become incorrect after retrieval, for KBPT training.Source: §A.1 · p. 12

A.2 Instructions

  • Medical reports are converted into closed-ended yes-or-no questions, then checked with GPT-4 and manually filtered.Manual filtering removes questions with obvious issues or those involving multiple images or patient histories.Source: §A.2 · p. 12

A.3 Involved Datasets

The appendix names three medical vision-language datasets and describes the evaluated open-source Med-LVLMs.

A.3 Involved Datasets

  • The datasets are MIMIC-CXR, IU-Xray, and Harvard-FairVLMed, covering chest X-rays and radiology reports as well as multimodal fundus data.Harvard-FairVLMed includes data from various sources and demographics, with a focus on fairness in multimodal fundus images.Source: §A.3 · p. 12

B Evaluated Models

  • The evaluation uses four open-source 7B-level models: LLaVA-Med, Med-Flamingo, MedVInT, and RadFM.Source: B Evaluated Models · p. 12
  • The model descriptions identify LLaVA-Med as a biomedical conversational assistant, Med-Flamingo as a medical few-shot learner, MedVInT as instruction-tuned, and RadFM as a radiology generalist for 2D and 3D scans.Source: B Evaluated Models · pp. 12–13

C Implementation Details

The experiments use CLIP-style vision and text encoders and train the model for 360 epochs. Hyperparameters are tuned by cross-validation, and retrieval uses reports from the corresponding dataset’s training set.

C Implementation Details

  • The vision encoder is ResNet-50 and the text encoder is bio-bert-based, following CLIP’s architecture and hyperparameter settings.The model uses AdamW with a learning rate of 10^-3, weight decay of 10^-2, and batch size 32, and is trained for 360 epochs.Source: C Implementation Details · p. 13
  • Reports available for retrieval come from the training set of the corresponding dataset, and hyperparameters are tuned by cross-validation with grid search.Source: C Implementation Details · p. 13
  • Experiments run on PyTorch 2.1.2 with four NVIDIA RTX A6000 GPUs; fine-tuning takes roughly 2.5 hours for CLIP and 4 hours for LLaVA-Med-1.5 7B.Source: C Implementation Details · p. 13

D Proofs

The proof defines the Med-LVLM and top-k retrieved contexts, then expresses factuality risk as an average over dataset examples. It uses Theorem 1 and Propositions 1 and 2 of Angelopoulos et al. (2021) to complete the proof.

D Proofs

  • The proof identifies M(·, ·) as the Med-LVLM and {T_k} as the top-k retrieved contexts.Source: D Proofs · p. 13
  • Each dataset example comprises a target image x_i, ground-truth answer y_i, and target question q_i.Source: D Proofs · p. 13
  • FR(k) is written as the average of a function evaluated at each data point in DMed, and the proof combines Theorem 1 and Propositions 1 and 2 of Angelopoulos et al. (2021).Source: D Proofs · p. 13
Loading 2407.05131v2…