PaperlayerView exact revision on arXiv

Source-linked AI summary

MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models

Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, Huaxiu Yao

arXiv:2410.13085v2cs.LGcs.CLcs.CV

TL;DR

Med-LVLMs can produce non-factual medical responses, while existing fine-tuning and RAG approaches face data, generalizability, and alignment challenges. MMed-RAG combines domain-aware retrieval, adaptive context selection, and preference fine-tuning. Across five medical datasets, it reports improved factual accuracy on both medical VQA and report generation.

  • Problem

    Med-LVLMs can produce non-factual medical responses, while fine-tuning is limited by scarce high-quality labels and distribution shifts and existing RAG methods are often dataset-specific and may introduce alignment issues.

    Source: §1 · p. 1

  • Method

    MMed-RAG combines domain-aware retrieval, adaptive selection of retrieved-context count, and RAG-based preference fine-tuning to improve cross-modality and ground-truth alignment.

    Source: §1 · p. 2

  • Results

    18.5% on medical VQA and 69.1% on report generation: factual-accuracy improvements over the original Med-LVLM across five medical multimodal datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMed-RAG improves factual accuracy across imaging domains, supporting its potential for reliable use in healthcare.

    Source: §7 · p. 10

    The theoretical claims are conditional: cross-modality analysis assumes a small initial image weight, while overall-alignment analysis assumes Lipschitz continuity with respect to retrieved context.

    Source: §4.1 · p. 6, §4.2 · p. 7

Abstract

from arXiv · show

Artificial Intelligence (AI) has demonstrated significant potential in healthcare, particularly in disease diagnosis and treatment planning. Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools. However, these models often suffer from factual hallucination, which can lead to incorrect diagnoses. Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to address these issues. However, the amount of high-quality data and distribution shifts between training data and deployment data limit the application of fine-tuning methods. Although RAG is lightweight and effective, existing RAG-based approaches are not sufficiently general to different medical domains and can potentially cause misalignment issues, both between modalities and between the model and the ground truth. In this paper, we propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an adaptive retrieved contexts selection method, and a provable RAG-based preference fine-tuning strategy. These innovations make the RAG process sufficiently general and reliable, significantly improving alignment when introducing retrieved contexts. Experimental results across five medical datasets (involving radiology, ophthalmology, pathology) on medical VQA and report generation demonstrate that MMed-RAG can achieve an average improvement of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available in https://github.com/richard-peng-xia/MMed-RAG.

1 INTRODUCTION

Med-LVLMs show promise for interactive diagnosis but can produce non-factual responses, a concern in clinical settings. MMed-RAG combines domain-aware retrieval, adaptive context selection, and preference fine-tuning to address factuality and alignment challenges.

1 INTRODUCTION

  • Med-LVLMs can generate non-factual medical responses, raising concerns about reliability in clinical settings where diagnostic errors can harm patient care.Fine-tuning is constrained by limited high-quality labeled data and training–deployment distribution gaps; existing RAG methods are often dataset-specific and may misalign modalities or model outputs with ground truth.Source: §1 · pp. 1–2
  • MMed-RAG selects retrieval models by image domain, adapts the number of retrieved contexts, and uses preference fine-tuning to improve cross-modality and ground-truth alignment.Its preference pairs discourage correct responses that ignore the image and encourage use of retrieved contexts when useful while avoiding irrelevant information.Source: §1 · p. 2
  • Across five datasets spanning radiology, pathology, and ophthalmology, MMed-RAG improves factual accuracy over the original Med-LVLM by 18.5% on medical VQA and 69.1% on report generation.The paper also reports theoretical mitigation of both alignment problems under mild assumptions.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries introduce medical vision-language model inputs and outputs, then outline preference optimization with DPO. They establish the notation used to describe model responses and preference data.

2 PRELIMINARIES

  • Med-LVLMs combine language models with medical visual modules, taking a medical image and clinical query as input and predicting the next token autoregressively.The text output is denoted y.Source: §2 · p. 2
  • DPO uses preference data to align a language-model policy, with preferred and dispreferred responses defined for an input prompt.The preference-pair probability is expressed using a sigmoid of the reward difference, and DPO formulates optimization as a classification loss over preference data.Source: §2 · p. 2
  • The reference policy πθ is the language model fine-tuned through supervised learning.It provides the reference policy used in the DPO formulation.Source: §2 · p. 2

3 MMED-RAG: A VERSATILE MEDICAL RAG SYSTEM

MMed-RAG combines domain-aware retrieval, adaptive context selection, and preference fine-tuning. Together, these modules choose retrieval sources and context amounts while training the model to use images and retrieved knowledge appropriately.

3.1 DOMAIN-AWARE RETRIEVAL MECHANISM

  • A domain identification module assigns each medical image a domain label and routes it to the corresponding multimodal retriever, which returns the top-k reports as references.The domain classifier is fine-tuned on labeled medical images, and retrievers are trained contrastively to align image and text embeddings.Source: §3.1 · p. 3

3.2 ADAPTIVE RETRIEVED CONTEXT SELECTION

  • Adaptive context selection sets k from retrieval similarity scores, truncating contexts when consecutive scores show a substantial relevance drop.The method uses the log ratio of consecutive scores and a fixed threshold; the paper notes that similarity scores vary with image complexity and textual alignment, unlike fixed-k selection.Source: §3.2 · p. 4

3.3 RAG-BASED PREFERENCE FINE-TUNING

  • RAG-based preference fine-tuning constructs pairs to favor image-grounded answers and helpful retrieval while discouraging interference from irrelevant contexts.The paper reports that RAG can yield correct answers despite a highly noisy replacement image in 55.08% of cases and can turn 43.31% of originally correct answers incorrect; its preference pairs target these alignment risks.Source: §3.3 · pp. 4–5

4 THEORETICAL ANALYSIS

The theoretical analysis derives conditional results for how preference losses change reliance on images and retrieved information. The results depend on assumptions about the reference model and the relationship between outputs and retrieved contexts.

4.1 THE IMPROVEMENT ON CROSS-MODALITY ALIGNMENT

  • Under Assumption 4.1, cross-modality loss increases the model’s weight on the image when its initial image weight is small.The paper explains that unrelated-image negative samples encourage the updated model to rely more on images than the initial model.Source: §4.1 · p. 6

4.2 THE IMPROVEMENT ON OVERALL ALIGNMENT

  • Under the overall-alignment assumptions, the loss increases reliance on useful retrieved context and decreases reliance on context associated with false answers.Assumption 4.2 requires Lipschitz continuity with respect to retrieved context; the paper’s analysis contrasts helpful retrieval with a distant, misleading alternative.Source: §4.2 · p. 7

5 EXPERIMENT

The experiments compare MMed-RAG with decoding, RAG, and open-source Med-LVLM baselines, then examine its component contributions and alignment behavior.

5.1 EXPERIMENTAL SETUPS

  • Experiments use LLaVA-Med-1.5 7B and five datasets for medical VQA and report generation, with pathology images excluded from report generation.The datasets cover radiology, ophthalmology, and pathology.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • MMed-RAG improves over the original Med-LVLM by 18.5% on medical VQA and 69.1% on report generation.The reported comparison covers the medical VQA and report-generation results.Source: Table 1 · p. 8, Table 2 · p. 8, §5.2 · p. 8
  • Against decoding-based methods, MMed-RAG improves by 11.5% and 44.2% on medical VQA and report generation; against RAG baselines, by 2.8% and 16.1%.The reported gains are larger for report generation in both comparisons.Source: §5.2 · p. 8
  • MMed-RAG outperforms open-source Med-LVLMs pretrained on large-scale datasets across medical image domains and multimodal tasks.The comparison reports average results across medical VQA and report generation.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 ANALYSIS

  • Across IU-Xray and Harvard-FairVLMed, domain-aware retrieval adds 17.9% and 16.1%, context selection 19.3% and 6.3%, and RAG-PT 37.1% and 16.9%.The reported component gains are listed in IU-Xray and Harvard-FairVLMed order.Source: Table 4 · p. 9, §5.3 · p. 9
  • Preference data tailored to cross-modality and overall alignment objectives positively affects model performance.The overall-alignment objective includes understanding retrieved knowledge and minimizing retrieval interference.Source: Table 5 · p. 9, §5.3 · p. 9
  • With MMed-RAG, Copy-Reference falls to 28.19%, while Over-Reliance falls from 43.31% to 8.38%.Figure 3 compares alignment with and without RAG.Source: §5.3 · p. 9, Figure 3 · p. 10
  • After RAG-PT, the model attends more to visual information and less to retrieved content, reducing RAG interference.The attention-map visualization highlights image-token attention enhanced by MMed-RAG.Source: Figure 4 · p. 10, §5.3 · p. 10

6 RELATED WORK

Related work describes factual hallucination in Med-LVLMs and existing medical RAG approaches, then identifies retriever generalizability and alignment as gaps addressed by MMed-RAG.

Factuality in Med-LVLMs

  • Med-LVLM outputs can contradict medical images, raising concerns about misdiagnoses or overlooked conditions (Xia et al., 2024a; Royer et al., 2024).Research specifically aimed at improving Med-LVLM factual accuracy remains relatively unexplored.Source: §6 · p. 10

Retrieval Augmented Generation in Med-LVLMs

7 CONCLUSION

The conclusion presents MMed-RAG as a multimodal RAG system for improving Med-LVLM factuality and alignment, with reported gains across imaging domains.

7 CONCLUSION

  • MMed-RAG combines domain-aware retrieval, adaptive selection of retrieved-context quantity, and RAG-based preference fine-tuning.The system is designed to improve cross-modality and ground-truth alignment.Source: §7 · p. 10
  • The paper reports improved Med-LVLM factual accuracy across radiology, ophthalmology, and pathology domains.The conclusion describes the results as supporting the system’s effectiveness across imaging domains.Source: §7 · p. 10
  • The authors frame reliable Med-LVLMs as a potential contribution to dependable tools in clinical settings.This is presented as the potential implication of the reported findings.Source: §7 · p. 10

ETHICS STATEMENT

The ethics statement describes the approach as RAG-based factuality enhancement and states that data handling and research conduct followed ethical and transparency principles.

ETHICS STATEMENT

  • The ethics statement describes the work as a RAG-based approach to improving Med-LVLM factuality.It expresses hope that the research will contribute to safer, more reliable AI-assisted medical tools.Source: ETHICS STATEMENT · p. 11
  • Datasets are described as publicly available or collected with appropriate ethical considerations, including patient-data anonymization.Source: ETHICS STATEMENT · p. 11
  • The authors state that they followed research-integrity, transparency, privacy, and regulatory principles.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The appendix documents the data, model, baseline, prompt, and training configurations used to make the experiments reproducible.

REPRODUCIBILITY STATEMENT

  • The reproducibility materials include system details, experimental configurations, and dataset processing steps, with algorithms documented in pseudocode.The paper points readers to Sections 3 and 5.1, Appendix A.1.2, Algorithm 1, and Appendices A.1.1 and A.2.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1.1 DATA STATISTICS

  • RAG-PT preference data comprises three sample types: correct answers retained under noisy images, corrected answers, and answers made incorrect by noise.The appendix distinguishes image-text pairs used for retriever training from QA items used for fine-tuning.Source: §A.1.1 · p. 16, Table 6 · p. 16, Table 7 · p. 16

A.1.2 HYPERPARAMETER SETTINGS

  • The retriever uses ResNet-50 and a BioBERT-based text encoder, while RAG-PT diffusion noise is controlled by ξ and reports come from the corresponding training set.The stated setup uses AdamW at 10^-4, batch size 512, and 360 epochs; hyperparameters are tuned with cross-validation and grid search.Source: §A.1.2 · p. 17

A.2 EVALUATED DATASETS

  • The five evaluated datasets cover chest radiography, fundus images, and pathology, including filtered PMC-OA pathology captions and Quilt-1M histopathology pairs.They are MIMIC-CXR, IU-Xray, Harvard-FairVLMed, PMC-OA, and Quilt-1M.Source: §A.2 · p. 17

A.3 EVALUATED MODELS

  • The evaluation uses five open-source 7B Med-LVLMs: LLaVA-Med, Med-Flamingo, MedVInT, RadFM, and miniGPT-Med.The appendix describes their distinct model designs and medical applications.Source: §A.3 · pp. 17–18

A.4 OVERVIEW OF THE BASELINES

  • Baselines include decoding methods that alter output-token logits and multimodal RAG systems, including MedDr, FactMM-RAG, and RULE.The decoding comparisons include Greedy Decoding, Beam Search, DoLa, OPERA, and VCD.Source: §A.4 · p. 18

A.5 PROMPTS

  • VQA questions are converted from medical reports into yes-or-no items, checked with GPT-4, then manually filtered for obvious issues and multi-image or patient-history questions.Prompt templates are provided in Table 9.Source: §A.5 · p. 19, Table 9 · p. 19

A.6.1 COMPATIBILITY ANALYSIS

MMed-RAG transfers to LLaVA-Med-1.0, improving its average performance and showing compatibility across backbones and domains.

A.6.1 COMPATIBILITY ANALYSIS

  • The reported transfer to LLaVA-Med-1.0 supports applying MMed-RAG beyond the LLaVA-Med-1.5 backbone used in the study.The authors describe the method as adaptable to various backbones.Source: §A.6.1 · p. 19
  • 40.3% average improvement over original LLaVA-Med-1.0; Table 10 reports performance on different backbones.The paper also reports consistent improvements across different Med-LVLMs and domains.Source: Table 10 · p. 19, §A.6.1 · p. 19

A.6.2 DETAILED RESULTS OF OTHER LVLMS

The appendix tests MMed-RAG across model families, medical domains, external data, and an environmental modeling task, while examining retrieval and alignment behavior. It reports broad improvements but also identifies settings where generalization or performance remains constrained.

A.6.2 DETAILED RESULTS OF OTHER LVLMS

  • MMed-RAG consistently outperforms compared general and medical LVLMs, including GPT-4o, Gemini-1.5, QwenVL, LLaVA-1.6, and InternVL-2.Table 11 presents the comparison.Source: §A.6.2 · p. 19

A.6.3 COMPARISON WITH DOMAIN-SPECIFIC MED-LVLMS AND THEM WITH RAG-PT

  • MMed-RAG outperforms domain-specific Med-LVLMs on radiology, pathology, and ophthalmology evaluations, and RAG-PT significantly improves those models.The comparisons use IU-Xray and PMC-OA pathology for VQA, and Harvard-FairVLMed for report generation.Source: §A.6.3 · p. 19, Table 12 · p. 20
  • Domain-specific models may outperform generalists in their specialties, yet RetinaVLM and RadFM remain below MMed-RAG even after RAG-PT.The authors suggest reduced ability to generalize to retrieved information as a possible constraint for some specialized models.Source: §A.6.3 · p. 20

A.6.4 RESULTS ON OTHER DOMAIN

  • RAG-PT also improves environmental modeling performance on CRW-Temp and CRW-Flow using a time-series encoder and text retriever.The evaluated model is LITE, an environmental system model based on LLaMA2.Source: §A.6.4 · p. 20, Table 13 · p. 21

A.6.5 STATISTICS OF COPY-REFERENCE RATE AND OVER-RELIANCE RATE FOR MORE LVLMS.

  • Open-source and commercial LVLMs show misalignment with retrieved information; GPT-4o has the best reported alignment, especially cross-modal.The alignment tests separately measure cross-modality alignment and overall alignment with ground truth.Source: §A.6.5 · p. 21, Table 14 · p. 21

A.6.6 DETAILED ABLATION ANALYSIS

  • Preference-data combination 1+3 yields the strongest targeted results, while using all three alignment objectives gives the best overall performance.The authors identify cross-modality and over-reliance issues as the most prominent pair of issues addressed by 1+3.Source: §A.6.6 · p. 21, Table 15 · p. 21

A.6.7 EXTERNAL VALIDATION

  • MMed-RAG significantly outperforms other Med-LVLMs on external fundus and microscopy validation subsets from PubMed-Vision.The authors present these same-domain external datasets as a check on overfitting and generalization.Source: §A.6.7 · p. 21, Table 16 · p. 22

A.6.10 COMPARISON UNDER FEW-SHOT SETTING

  • The appendix reports that LLaVA-Med-1.5 performance decreases in the tested few-shot setting, while domain-aware retrieval reaches 99.83% modality-identification accuracy.The retriever comparison favors the proposed retriever on radiology image-to-text retrieval; rationale-guided RAG is also reported to further improve performance.Source: §A.6.10 · p. 23, §A.6.11 · p. 23, §A.6.12 · p. 23

B.3 PROOFS

The proofs characterize how preference optimization shifts model probabilities and linear-model coefficients. They also show that cross-modality and the overall loss change the weights assigned to specified inputs.

B.3 PROOFS

  • Lemma B.1 shows optimization increases πo(y|x) when qw(y|x) > ql(y|x), which is more likely for preferred samples given xt and xr.The lemma is proved using Lemmas B.2 and B.3.Source: §B.3 · pp. 26–27
  • Equation 25 holds if and only if f(xv, xt, xr, y) equals g(xt, xr) plus log qw(xv, y|xt, xr) for some g.Source: §B.3 · p. 27
  • Lemma B.3 provides a closed-form solution to equation 14 when the parameter space is sufficiently large, supporting the proof of Lemma B.1.Source: §B.3 · p. 27
  • For linear-Gaussian DPO, θo moves toward β for preferred responses and away from β̃ for dispreferred responses.The corollary assumes independent standard-normal errors for the preferred, dispreferred, and model responses.Source: §B.3 · p. 28
  • Under Assumption B.2, cross-modality increases the weight of xv.Source: §B.3 · p. 29
  • Under Assumption B.3, the overall loss increases the weight of xr and decreases the weight of ˜xr.Source: §B.3 · p. 29
Loading 2410.13085v2…