PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Retrieval-Augmented Generation for Medicine

Guangzhi Xiong, Qiao Jin, Zhiyong Lu, Aidong Zhang

arXiv:2402.13178v2cs.CLcs.AI

TL;DR

Medical RAG lacks clear best practices for selecting corpora, retrievers, and backbone LLMs, despite hallucinations and outdated knowledge posing challenges in medical QA. The paper introduces MIRAGE and MEDRAG to benchmark these choices across medical QA settings, finding that MEDRAG improves LLM accuracy by up to 18% over chain-of-thought prompting and supports practical recommendations. The study primarily evaluates vanilla RAG and leaves newer architectures and additional resources for future work.

  • Problem

    Medical LLMs face hallucinations and outdated knowledge, while best practices for tuning the flexible components of medical RAG systems remain unclear.

  • Method

    The paper introduces MIRAGE, a 7,663-question benchmark from five medical QA datasets, and MEDRAG, a toolkit spanning corpora, retrievers, and LLMs.

  • Results

    Across different LLMs, MEDRAG yields 1% to 18% relative performance increases over chain-of-thought prompting, with GPT-3.5 and Mixtral reaching performance comparable to GPT-4 on MIRAGE.

  • Takeaways & Limitations

    The evaluations provide practical recommendations for researching and deploying medical RAG systems.

  • Takeaways & Limitations

    The study mainly evaluates vanilla RAG and does not cover newer RAG architectures or all potentially useful resources.

Abstract

from arXiv · show

While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the "lost-in-the-middle" effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.

1 Introduction

The paper introduces MIRAGE and MEDRAG to systematically evaluate medical RAG systems, addressing hallucinations, outdated knowledge, and unclear component-selection practices. Experiments show that MEDRAG improves multiple LLMs and supports practical recommendations for medical RAG.

  • 7,663 questions from five biomedical QA datasets support zero-shot, question-only retrieval evaluation across five corpora, four retrievers, and six LLMs.
  • 1% to 18% relative performance increases over chain-of-thought prompting were observed across different LLMs using MEDRAG.
  • PubMed supports all MIRAGE tasks, while BM25 and MedCPT perform strongly and combining retrievers can further enhance performance.
  • The study reports log-linear scaling, a “lost-in-the-middle” phenomenon, and practical recommendations for biomedical RAG.
  • MIRAGE is a benchmark for systematically comparing different medical RAG systems.
  • MEDRAG is a medical QA toolkit covering domain-specific corpora, retrievers, and LLMs.

2 Related Work

Prior work established RAG for grounding and updating LLM knowledge, while biomedical applications remained incomprehensively evaluated. This paper evaluates medical QA with both RAG and question-only retrieval settings.

  • RAG integrates retrieved relevant information to mitigate hallucinations and provide knowledge not encoded by LLMs.
  • Biomedical RAG has been explored for literature information-seeking and clinical decision-making, but prior evaluations were not comprehensive.
  • Medical QA datasets evaluate biomedical capabilities of general and domain-specific LLMs because medical QA is knowledge-intensive.
  • Unlike prior efforts, this evaluation uses both RAG and question-only retrieval settings for medical QA.

3 The MIRAGE Benchmark

MIRAGE is designed as a scalable, realistic benchmark for medical information needs, combining zero-shot, multiple-choice, RAG, and question-only retrieval settings. It contains five medical QA datasets and evaluates answer-choice accuracy.

  • 3.1 Evaluation Settings: MIRAGE adopts zero-shot learning, multiple-choice evaluation, RAG, and question-only retrieval as four evaluation settings.
  • 3.1 Evaluation Settings: Question-only retrieval excludes answer options from retrieval input to better reflect real-world medical QA.
  • 3.2 Component Datasets: MIRAGE includes five medical QA datasets: MMLU-Med, MedQA-US, MedMCQA, PubMedQA*, and BioASQ-Y/N.
  • 3.2 Component Datasets: The benchmark contains three medical examination datasets and two biomedical research datasets, using only multi-choice questions and excluding supporting contexts.
  • 3.3 Evaluation Metric: Each task is evaluated by answer-choice accuracy, with standard deviation reported and average accuracy across five tasks summarizing general medical QA performance.

4 The MEDRAG Toolkit

MEDRAG is a modular toolkit for medical QA that combines multiple corpora, retrievers, and backbone LLMs. It supports systematic comparison of these components, including domain-specific and general-purpose options.

  • MEDRAG consists of three major components: corpora, retrievers, and LLMs.
  • Corpora: The toolkit combines PubMed, StatPearls, medical Textbooks, and Wikipedia into separate sources or the unified MedCorp corpus.Each corpus is divided into short snippets for retrieval.
  • Retrievers: MEDRAG includes BM25, Contriever, SPECTER, and MedCPT as lexical, general-domain, scientific-domain, and biomedical-domain retrievers.
  • LLMs: The evaluated LLMs include GPT-3.5, GPT-4, Mixtral, Llama2, MEDITRON, and PMC-LLaMA.Retrieved snippets are concatenated and prepended to each question input.

5 Results

The results show that MEDRAG performance depends on the backbone LLM, corpus, retriever, and their combination. Retrieval generally improves medical QA, but its gains vary by task and configuration, with corpus selection and retrieval fusion strongly affecting outcomes.

  • 5.1 Comparison of Backbone LLMs: GPT-4 achieves 73.44% average accuracy on MIRAGE with chain-of-thought prompting.Other backbone LLMs reach about 70% with MEDRAG, compared with about 61% under chain-of-thought prompting.
  • 5.1 Comparison of Backbone LLMs: MEDITRON and PMC-LLaMA outperform other models on PubMedQA* under chain-of-thought prompting by +34.8% and +40.9% relative gains over GPT-4.MEDITRON also exceeds its base Llama2 model by +13.5% with chain-of-thought prompting and +12.7% with MEDRAG.
  • 5.1 Comparison of Backbone LLMs: +19.4% and +20.9% are Llama2’s MEDRAG improvements on PubMedQA* and BioASQ-Y/N, respectively.These gains produce performance comparable to or better than MEDITRON with chain-of-thought prompting.
  • 5.1 Comparison of Backbone LLMs: MEDRAG does not always improve over supervised fine-tuning on examination questions because helpful snippets may be difficult to retrieve.The contrast with literature-based questions indicates remaining room for improvement.
  • 5.2 Comparison of Corpora and Retrievers: 76.68% is GPT-3.5 MEDRAG accuracy with Textbooks on MMLU-Med, while 67.48% is its accuracy with StatPearls on MedQA-US.PubMed is the only corpus reported to improve all MIRAGE tasks, whereas MedCorp performs robustly across tasks.
  • 5.2 Comparison of Corpora and Retrievers: MedCorp improves almost all retrievers on MedMCQA by -1.5% to +5.0%, even though no single corpus helps that task.
  • 5.2 Comparison of Corpora and Retrievers: RRF-4 with MedCorp increases average performance by 1.4% to 10.7% over individual retrievers.On MedCorp, RRF-4 reaches a 71.57% state-of-the-art average score on MIRAGE.

6 Discussions

The evaluation shows that medical RAG performance depends on snippet count, snippet position, corpus, retriever, and language-model choices. These findings support task-specific configuration recommendations for medical QA.

  • Snippet scaling: RAG performance generally improves as the number of snippets increases, but can decline when excessive snippets reduce the signal-to-noise ratio.For k ≤8, retrieved information may be insufficient; performance can drop when k becomes too large.
  • Snippet scaling: PubMedQA* reaches high accuracy with k = 1, while additional snippets can hurt performance because more irrelevant documents are included.The retriever identifies 79.6% of ground-truth snippets as the top-1 context for PubMedQA*.
  • Snippet position: Accuracy follows a U-shaped decreasing-then-increasing pattern as the ground-truth snippet moves through the context.This pattern informs future arrangements of snippets in medical RAG systems.
  • Corpus composition: Retrieved corpus composition is task-specific: examination questions favor Textbooks and StatPearls, whereas research questions favor PubMed.Wikipedia generally occupies a smaller share of retrieved medical snippets than of MedCorp because many Wikipedia snippets are not biomedical.
  • Corpus selection: PubMed and MedCorp are the only corpora reported to outperform chain-of-thought prompting across all MIRAGE tasks.MedCorp is described as a more comprehensive and reliable choice when resources permit, while other single corpora may help specific examination tasks.
  • Retriever selection: MedCPT is the most reliable individual retriever, while BM25 is also strong and retriever fusion can provide robust performance when chosen carefully.RRF-2 combining BM25 and MedCPT is recommended for PubMed; RRF-2 and RRF-4 are reliable choices for MedCorp.
  • LLM selection: GPT-4 achieves about 80% accuracy on MIRAGE, while GPT-3.5 offers a more cost-efficient alternative and Mixtral can run locally and offline.The recommendations distinguish accuracy, cost efficiency, and privacy-oriented deployment considerations.

7 Conclusion

The paper introduces MIRAGE and MEDRAG to evaluate medical RAG systems systematically. Its evaluations produce observations and practical recommendations for biomedical RAG research and deployment.

  • MIRAGE and MEDRAG provide a benchmark and toolkit for evaluating medical RAG systems.The paper presents the resulting observations and recommendations as guidance for research and real-world deployments.

Limitations

The study evaluates a widely used vanilla RAG architecture but does not cover all newer designs, resources, or retrieval analyses. These boundaries leave several directions for future work.

  • The study mainly evaluates vanilla RAG, in which retrieved documents are directly prepended to the LLM context.Newer architectures such as active RAG remain important directions for future evaluation.
  • MEDRAG does not include potentially useful resources such as full-text PubMed Central articles and trustworthy-source FAQs.The authors identify these resources as candidates for future incorporation.
  • Retrieval is evaluated only for PubMedQA* and BioASQ-Y/N because the three examination datasets lack ground-truth supporting-document labels.Future work should assess snippet helpfulness for examination datasets and examine cross-encoder rerankers.

A Details of MIRAGE Datasets

MIRAGE combines five medical QA datasets spanning multitask, professional-exam, biomedical research, and biomedical yes/no questions. The datasets differ in size, format, source, and availability of supporting contexts.

  • MMLU-Med: MMLU-Med uses six biomedically relevant tasks from the 57-task MMLU benchmark.The selected areas include anatomy, clinical knowledge, professional medicine, human genetics, college medicine, and college biology.
  • MedQA-US: MedQA-US contains 1,273 four-option questions from the English portion of the US Medical Licensing Examination.
  • MedMCQA: MedMCQA contributes 4,183 medical questions from its development set because test-set ground truth is unavailable.The original dataset covers about 2.4k healthcare topics and 21 medical subjects.
  • PubMedQA*: PubMedQA* contains 500 expert-annotated test questions with their original contexts removed to test retrieval and answering.The source dataset consists of biomedical research questions constructed from PubMed abstracts.
  • BioASQ-Y/N: BioASQ-Y/N includes 618 yes/no questions selected from the ground-truth test sets of BioASQ Task B from 2019–2023.BioASQ includes both information-retrieval and machine-reading-comprehension tracks.

B.1 Document Collections

MEDRAG uses multiple biomedical and general-knowledge corpora, including PubMed, StatPearls, medical textbooks, and Wikipedia, with snippets retrieved by lexical, dense, and fused systems.

  • Document Collections: PubMed contributes a 23.9-million-article subset containing biomedical titles and abstracts with valid metadata.The broader PubMed resource contains over 36 million biomedical articles.
  • Document Collections: StatPearls contributes 9,330 publicly available point-of-care clinical articles structured into paragraph snippets with hierarchical headings.The paper describes this as the first evaluation of StatPearls in biomedical NLP.
  • Retrieval Systems: BM25 provides lexical TF-IDF retrieval, while Contriever, SPECTER, and MedCPT provide dense retrieval based on general, scientific, or biomedical representations.MedCPT is contrastively pretrained on 255 million PubMed search-log user clicks and achieves state-of-the-art performance on several biomedical information-retrieval tasks.
  • Retrieval Systems: RRF-2 fuses BM25 with MedCPT, whereas RRF-4 combines all individual retrievers used in MEDRAG.These fusion systems merge retrieval results from different sources through shared predictions.

B.3 Backbone LLMs

MEDRAG evaluates GPT-3.5, GPT-4, Mixtral, Llama2, MEDITRON, and PMC-LLaMA, using standard prompts for most models and a pseudo one-shot demonstration for MEDITRON.

  • Backbone LLMs: MEDRAG includes GPT-3.5-turbo-16k-0613 and GPT-4-32k-0613 accessed through Microsoft Azure OpenAI Services.Both are commercial LLMs used for medical question answering.
  • Backbone LLMs: The open-source backbone set includes Mixtral-7×8B, Llama2-70B, MEDITRON-70B, and PMC-LLaMA-13B.These models span sparse mixture-of-experts, general instruction-tuned, biomedical, and PubMed Central-fine-tuned systems.
  • Prompt Templates: MEDITRON receives a pseudo one-shot demonstration because its released checkpoint is pretrained only and does not follow the system prompt well.Separate CoT and MEDRAG templates are provided for MEDITRON.
  • Prompt Templates: CoT prompts ask models to reason step-by-step before selecting an answer choice and returning a structured JSON response.The prompt templates apply this format across the evaluated medical multiple-choice questions.
  • Prompt Templates: MEDRAG prompts add relevant retrieved documents to the same step-by-step medical QA format before answer selection.The template explicitly instructs the model to use the relevant documents.
Loading 2402.13178v2…