PaperlayerView exact revision on arXiv

Source-linked AI summary

M3Retrieve: Benchmarking Multimodal Retrieval for Medicine

Arkadeep Acharya, Akash Ghosh, Pradeepika Verma, Kitsuchart Pasupa, Sriparna Saha, Priti Singh

arXiv:2510.06888v1cs.IRcs.AI

TL;DR

Medical retrieval lacks a standardized benchmark for evaluating multimodal models, despite the use of both images and text in healthcare. M3Retrieve provides a medical benchmark and evaluates retrieval models across specialties. The evaluation identifies discipline-specific challenges, including strong model performance on particular tasks.

  • Problem

    Medical applications lack a standardized benchmark for evaluating multimodal retrieval models, although medical information often combines images and text.

    Source: §1 · pp. 1–2

  • Method

    M3Retrieve integrates textual and visual data in a medical retrieval benchmark and evaluates models across medical specialties.

    Source: §1 · pp. 2–3

  • Results

    Evaluation reveals discipline-specific retrieval challenges; in Visual Context Retrieval, MM-Embed achieves the highest NDCG@10 score of 45.47.

    Source: §1 · p. 3, §5.1 · p. 8

  • Takeaways & Limitations

    The findings underscore the need for a medical-specific multimodal retrieval model because existing models underperform in medical retrieval compared with general-domain tasks.

    Source: §8 · p. 9

    The benchmark covers 16 broad medical domains, which may not encompass all specialties.

    Source: §8 · p. 9

Abstract

from arXiv · show

With the increasing use of RetrievalAugmented Generation (RAG), strong retrieval models have become more important than ever. In healthcare, multimodal retrieval models that combine information from both text and images offer major advantages for many downstream tasks such as question answering, cross-modal retrieval, and multimodal summarization, since medical data often includes both formats. However, there is currently no standard benchmark to evaluate how well these models perform in medical settings. To address this gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark. M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over 1.2 Million text documents and 164K multimodal queries, all collected under approved licenses. We evaluate leading multimodal retrieval models on this benchmark to explore the challenges specific to different medical specialities and to understand their impact on retrieval performance. By releasing M3Retrieve, we aim to enable systematic evaluation, foster model innovation, and accelerate research toward building more capable and reliable multimodal retrieval systems for medical applications. The dataset and the baselines code are available in this github page https://github.com/AkashGhosh/M3Retrieve.

1 Introduction

Medical retrieval lacks a standardized benchmark that captures clinical specialties and image–text relationships. M3Retrieve addresses this gap with multimodal retrieval tasks and broad medical coverage.

1 Introduction

  • Existing benchmarks lack the scale and diversity for fine-grained assessment across medical disciplines, and M-BEIR datasets do not explicitly cover medical text-image pairs.Medical retrieval also involves complex terminology, specialized disciplines, and image-text relationships that can depend on patient history.Source: §1 · pp. 1–2
  • M3Retrieve is presented as a large-scale benchmark that accepts multimodal queries and targets realistic document stores across multiple specialties.The benchmark integrates textual and visual data to evaluate retrieval in complex multimodal medical contexts.Source: §1 · pp. 2–3
  • The benchmark aggregates 22 curated datasets and covers 16 medical disciplines, with 920 K text documents and 818 K multimodal queries.The datasets are described as permissively licensed and intended to provide broad coverage of real-world clinical scenarios.Source: §1 · p. 3
  • The clinically grounded task suite covers Visual Context Retrieval, Query-to-Image Retrieval, Case Study Retrieval, and Multimodal Summarisation Retrieval.The described tasks use combinations of images, text, reports, cases, and summaries as queries or retrieval targets.Source: §1 · p. 3
  • Evaluation of state-of-the-art multimodal retrieval models reveals discipline-specific challenges and quantifies their impact on retrieval effectiveness.The introduction frames this benchmark as the first large-scale multimodal retrieval benchmark for medicine.Source: §1 · p. 3

2 Related Works

Prior retrieval benchmarks and model architectures provide a foundation, but existing benchmarks do not fully address medical retrieval’s clinical detail and multimodal demands. M3Retrieve frames a five-task benchmark for evaluating these models in healthcare.

2.1 Retrieval Benchmarks

  • UniIR and other retrieval benchmarks do not comprehensively address medical clinical detail, terminology, and combined visual-textual evidence, leaving a need for a dedicated medical benchmark.Earlier medical benchmarks largely focus on text retrieval and question answering, while nontextual data is rarely incorporated.Source: §2.1 · pp. 3–4

2.2 Retrieval Models

  • Retrieval models have progressed from lexicon-based and dense systems to multimodal and late-interaction architectures, including models for medical image-text retrieval.The related-work discussion presents M3Retrieve as a means to assess discipline-specific challenges and multimodal signal integration, supporting practical guidelines for reliable medical retrieval.Source: §2.2 · p. 4

3 Creation of the M3Retrieve Benchmark

This section describes task-specific dataset construction and relevance mapping for multimodal medical retrieval, followed by expert review of a sample of queries.

3.1 VisualContext Retrieval

  • VisualContext Retrieval ranks relevant corpus documents for a multimodal query, drawing on Wikipedia, PubMed, and open-access medical textbooks.Source: §3.1 · p. 5, §3.1.1 · p. 5

3.1.1 Data Curation and Relevance Design

  • Textbook relevance includes the figure-linked paragraph and section context; Wikipedia uses figure-linked paragraphs plus article context, while PubMed retains only figure-referencing paragraphs.Source: §3.1.1 · p. 5
  • A unified scoring framework across data sources was used to create a consistent benchmark for multimodal evidence retrieval.Source: §3.1.1 · p. 5

3.2.1 Data Curation and Relevance Design

  • Multimodal Summary Retrieval matches clinical text-image contexts to summaries; its corpus combines 3,015 MMQS samples with 228,887 HealthcareMagic query-summary records.Source: §3.2 · p. 5, §3.2.1 · pp. 5–6
  • For summary retrieval, the summary corresponding to a multimodal query receives relevance score 2, while all other corpus summaries receive score 0.Source: §3.2.1 · p. 6

3.3.1 Data Curation and Relevance

  • Multimodal query-to-image retrieval uses MedPix 2.0, treating images from the query’s clinical case as relevant and assigning them score 2.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.4.1 Data Curation and Relevance

  • Case-study retrieval pairs multimodal clinical queries with structured cases, using MultiCaRe narratives and images linked by case_id.Same-record cases score 2, related diagnoses or overlapping symptoms from other records score 1, and remaining cases score 0; MultiCaRe contains over 93,000 cases and more than 130,000 diagnostic images.Source: §3.4 · p. 6, §3.4.1 · pp. 6–7

3.5 Quality Control Using Domain Expert

  • 80 queries per task were reviewed by two doctors, with Cohen’s kappa of 0.78, supporting agreement on the dataset rankings.Source: §3.5 · p. 7

4 Experimental Setup

The experiments use nNDCG@10 as the primary metric and compare lexical, text-based, CLIP-style, and multimodal retrieval baselines.

4 Experimental Setup

  • Experiments used the MTEB Python library on NVIDIA A100 80GB GPUs, with nNDCG@10 as the primary evaluation metric.Source: §4 · p. 7

4.1 Baseline Models

  • The baseline suite includes BM25, E5-Large-v2, BGE-en-Large, and NV-Embed-v2, alongside CLIP-style and multimodal encoders.Source: §4.1 · p. 7
  • FLMR provides a multimodal late-interaction baseline using token-level similarity between queries and documents.Source: §4.1 · p. 7

5 Results Analysis

Across the four tasks, multimodal models lead when text and images must be integrated, while uni-modal dense retrievers lead on text-centric tasks. The analysis also compares five retrieval paradigms and notes that results depend on dataset structural assumptions.

5 Results Analysis

  • The evaluation covers lexicon-based, text-based dense, CLIP-style, multimodal encoder, and late-interaction models, with results reported across four tasks.The authors caution that reported results are grounded in the underlying datasets' structural assumptions.Source: §5 · p. 7

5.1 Overall Performance Trends

  • 45.47 NDCG@10 for MM-Embed in VisualContext Retrieval and 43.53 for MedImageInsight in Query to Image Retrieval; NV-Embed leads Summary Retrieval at 89.73 and Case Study Retrieval at 10.99.The reported pattern favors multimodal models for tasks integrating text and images, and uni-modal dense retrievers for predominantly textual tasks.Source: Table 3 · p. 8, §5.1 · p. 8

5.2 Task-Specific Model Performance

  • 38.07 NDCG@10 for BM25 and 43.28 for NV-Embed follow MM-Embed's 45.47 in VisualContext Retrieval; MM-Embed scores 29.49 and CLIP SF 29.06 behind MedImageInsight's 43.53 in Query to Image Retrieval.The text describes VisualContext Retrieval as benefiting from multimodal integration and Query to Image Retrieval as favoring CLIP-style models.Source: §5.2 · p. 8
  • 83.66 for BGE and 70.23 for E5 Large follow NV-Embed's 89.73 in Summary Retrieval; MM Ret scores 10.87 behind NV-Embed's 10.99 in Case Study Retrieval.The authors say current multimodal models lag on primarily textual Summary Retrieval, while MM Ret shows some potential for Case Study Retrieval.Source: §5.2 · p. 8

6 Conclusion

The authors present M3Retrieve as a multimodal medical retrieval benchmark spanning more than 16 domains and 500K+ documents. They say it provides detailed evaluation of retrieval approaches and a foundation for developing more effective specialized systems.

6 Conclusion

  • M3Retrieve is presented as the first comprehensive multimodal medical retrieval benchmark, covering 16+ medical domains and 500K+ documents.It assesses uni-modal and multimodal models across five retrieval styles and architectural approaches.Source: §6 · p. 8
  • The benchmark provides a detailed analysis of retrieval approaches in complex medical scenarios and helps identify current models' gaps and strengths.Source: §6 · p. 8
  • The authors believe M3Retrieve will help drive research and advance medical AI systems for real-world healthcare applications.Source: §6 · pp. 8–9

7 Ethical Considerations

M3Retrieve uses publicly available CC-licensed data, with medical professionals involved in dataset design and a post-construction human evaluation of quality and fairness.

7 Ethical Considerations

  • All M3Retrieve data come from publicly available sources with Creative Commons licenses, which the authors say ensures HIPAA and GDPR compliance.Source: §7 · p. 9
  • Medical professionals were involved throughout dataset design to support relevance and accuracy.Source: §7 · p. 9
  • A post-construction human evaluation was conducted to verify benchmark quality and fairness.Source: §7 · p. 9

8 Limitation and Future Work

The benchmark's coverage and validation have limits, and its current query-document setup pairs multimodal queries with text-only documents. The authors identify expanding specialty coverage and modalities, as well as broader mapping verification, as future work.

8 Limitation and Future Work

  • Coverage of 16 broad medical domains may omit specialties, and future work will expand the benchmark to additional disciplines.Source: §8 · p. 9
  • Current multimodal queries are paired with text-only documents; future versions are intended to add modalities to both queries and corpora.Source: §8 · p. 9
  • Existing models underperform in medical retrieval compared with general-domain tasks, motivating the authors' call for a medical-specific multimodal retrieval model.Source: §8 · p. 9
  • Figure-to-paragraph mappings received only limited manual validation, since large-scale verification across the dataset was not feasible.The approach therefore relies on the assumption that figures are responsibly referenced in the original publications.Source: §8 · p. 9

A.1 Example of datapoints in the M3Retrieve Benchmark

Figure 3 shows a query image-text pair and corpus texts alongside justifications for assigned scores.

  • Figure 3 shows a query image-text pair and corpus texts alongside justifications for assigned scores.Source: §A.1 · p. 11

A.2 Discipline wise analysis of the tasks in the M3Retrieve Benchmark

Across Visual Context Retrieval, MM Embed outperforms unimodal models, although dense encoders remain below their BEIR averages. Case Study Retrieval and query-to-image results vary substantially by specialty and anatomical domain.

A.2.1 Visual Context Retrieval

  • 66.29 for NV Embed and 61.20 for BM25 lead in Anatomy and Physiology; MM Embed scores 50.28 in Psychiatry and Mental Health and 70.06 in PubMed Retrieval, versus BM25 at 68.72.Orthopedics and Musculoskeletal scores remain low, including BM25 at 6.27 and NV Embed at 6.86.Source: §A.2.1 · p. 11
  • MM Embed outperforms all unimodal models, while NV Embed and MM Embed show 30% and 25.37% lower NDCG@10 scores than their BEIR averages (62.65 and 60.3).Dense encoders outperform BM25 and CLIP-style models overall, while BM25 remains strong in text-heavy disciplines.Source: §A.2.1 · p. 11

A.2.2 Case Study Retrieval

  • 14.21 for BM25 in Orthopedics, 13.51 in Pathology, and 13.08 in Gastroenterology show its strength; E5 and BGE also perform strongly in Gastroenterology and Genetics and Genomics.NV Embed scores 13.59 in Genetics and Genomics.Source: §A.2.2 · pp. 11–12
  • MII averages 1.64 in improvements and peaks at 2.31 in Cardiology, while FLMR reaches 2.45 in Genetics and Genomics and 2.05 in Gastroenterology.CLIP SF excels in Rheumatology and Immunology (1.85) and Cardiology (2.08); BLIP FF adds lift in Dermatology (1.17).Source: §A.2.2 · pp. 12–13

Domain-wise Model Performance Analysis

  • MII leads in Spine and Muscles (37.88), Abdomen (49.04), Head (34.31), Thorax (47.27), and Reproductive and Urinary System (49.14).CLIP SF and MM Embed also score strongly in several domains, including 29.41 and 27.54 in Abdomen, and 35.03 and 33.75 in Thorax.Source: Domain-wise Model Performance Analysis · p. 13
  • 43.53 is MII’s highest average across anatomical domains; CLIP SF scores 29.12 and MM Embed 29.49, while BLIP FF, MM Ret, and FLMR average 2.07, 2.28, and 2.56.Source: Domain-wise Model Performance Analysis · p. 13
Loading 2510.06888v1…