MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval

• Expert-domain applications such as medical decision support depend on retrieving and integrating external information across visual and textual modalities.Source: §1 · p. 1
• Prior benchmarks largely focus on general-domain knowledge, semantic matching, or information seeking, while real-world expert queries may require deeper image interpretation and interleaved text with multiple images.Source: §1 · pp. 1–2
• MRMR contains 1,435 expert-annotated examples across 23 domains and three task types: Knowledge, Theorem, and Contradiction, with queries and documents represented in interleaved multimodal format.Source: §1 · p. 2
• 54.1 nDCG@10 is the highest performance, achieved by Qwen3-Embedding with LLM-based image captioning; multimodal systems underperform text-only captioning systems on knowledge- and reasoning-intensive queries.Source: §1 · p. 2
• Table 1 compares multimodal retrieval benchmarks by domain count and whether they require expertise, intensive reasoning, and interleaved image-text data.Source: Table 1 · p. 3

• Prior benchmarks cover paired image-text matching, composed image retrieval, and evidence or document retrieval, but generally use single-image inputs; TIIR supports interleaved sequences yet is limited to general-domain WikiHow tutorials.Source: §2 · p. 3
• Multimodal retrieval models use shared image-text encoders, fusion strategies, or MLLM embeddings; many MM-RAG benchmarks assess generation without evidence-level relevance annotations.Source: §2 · p. 3
• Table 2 reports dataset query and document counts, positive-document averages, text lengths, image counts, and sources; Knowledge and Theorem datasets each share a retrieval corpus.Source: Table 2 · p. 4
• Text-domain reasoning-intensive retrieval benchmarks target documents that support problem-solving through shared theorems; MRMR extends this retrieval setting to multimodal queries and interleaved web documents.Source: §2 · p. 4

3.1 TASK FORMULATION

• 3.1 TASK FORMULATION: Queries and documents are text-image segment sequences; relevance may depend on expert knowledge, a theorem needed for calculation, or a rule that conflicts with a case.Source: §3.1 · pp. 4–5

3.2 KNOWLEDGE: RETRIEVING WEB PAGES THAT HELP ANSWER QUESTIONS

• 3.2 KNOWLEDGE: MMMU-Pro knowledge questions become queries; web documents are captured, OCR-processed, chunked, screened by GPT, and expert-verified, while PIN-14M supplies additional negatives.Source: §3.2 · p. 5, Figure 2 · p. 6

3.3 THEOREM: RETRIEVING RELEVANT THEOREMS THAT SOLVE PROBLEMS

• 3.3 THEOREM: Image-centric calculation problems are matched to applicable theorem statements, using BRIGHT documents and additional theorem sources when relevant counterparts are unavailable.Source: §3.3 · p. 6

3.4 CONTRADICTION: RETRIEVING CONTRADICTORY RULES AND REQUIREMENTS

• 3.4 CONTRADICTION: A case description is matched to a mandated rule or requirement it violates, demanding logical conflict detection beyond semantic matching.Source: §3.4 · pp. 6–7

Negation.

• Negation.: The synthetic task pairs COCO images with four candidate descriptions, one contradicting the image by asserting a missing object or denying an existing one.Source: Negation · p. 7

Vehicle Design.

• Vehicle Design.: Formula SAE requirements are matched against design cases to retrieve specifications the vehicle fails to satisfy, such as a minimum wheelbase requirement.Source: Vehicle Design · p. 7

Traffic Case.

• Traffic Case.: The task tests whether retrievers identify violated driving rules by matching annotated traffic cases, augmented with AI-generated images, to handbook sections.Source: Traffic Case · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation covers four retrieval setups: captioned text retrievers, two-stream vector fusion, merged-image multimodal models, and document-as-image retrieval.The study evaluates 14 models; it uses nDCG@10 except for Negation, where it uses Hit@1.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

• 54.1 nDCG@10: Qwen3-Embedding with LLM-based image captioning outperforms the evaluated multimodal systems, with Ops-MM-Embedding trailing by 6.0 points.Table 3 reports performance across 11 subtasks, using Hit@1 for Negation and nDCG@10 for the others.Source: §4.2 · p. 8, Table 3 · p. 8
• 67.4 nDCG@10 on Knowledge falls to 37.4 on Theorem and 36.6 on Contradiction for Ops-MM-Embedding, while most models score below 25% Hit@1 on Negation.The Negation score is equivalent to random guessing among four candidates; E5-V and VLM2Vec also perform poorly on reasoning-intensive tasks.Source: §4.2 · p. 8
• Current systems can match semantically relevant documents without resolving conflicts, but remain limited in reasoning ability.Even the smaller Design and Traffic candidate corpora do not prevent models from struggling to identify underlying contradictions.Source: §4.2 · p. 9
• 8.6 versus 48.1 nDCG@10: E5-V and Ops-MM-Embedding, respectively, show wide performance variation among merged-image multimodal models.MM-Embed is competitive with Ops-MM-Embedding in Clinical Medicine and Diagnostics but lags in art-related tasks; visual overlap is more common in art than medical imaging.Source: §4.2 · p. 9

5.1 QUALITATIVE ANALYSIS

• Across 30 manually reviewed top-10 retrieval case studies, Ops-MM-Embedding sometimes ranked visually similar negatives above positives, including failures to infer a car crossing a lane line contrary to the positive instruction.Similar errors occurred with eye images from different diseases; in the Traffic case, both documents depicted cars, tunnels, and lane markings.Source: §5.1 · p. 9

5.2 TEST-TIME SCALING IN RETRIEVAL

• Replacing queries with MLLM-generated reasoning traces improves performance by 5.1 for Qwen2-VL-2B and 14.8 for Qwen2.5-VL-72B, with larger gains especially on Knowledge tasks.Without output-length constraints, Qwen2.5-VL-72B produces 20% more tokens on Knowledge and 66% more on Theorem than Qwen2-VL-2B, trading higher inference cost for larger gains.Source: §5.2 · p. 9

6 CONCLUSION

• MRMR combines knowledge- and reasoning-intensive questions from MMMU-Pro with a multimodal corpus whose positive documents are verified by human experts.Source: §6 · p. 10
• Contradiction Retrieval evaluates whether models can identify conflicts, complementing the benchmark’s knowledge and reasoning-intensive retrieval focus.Source: §6 · p. 10
• The evaluation finds multimodal retrieval systems lagging behind text-retrieval counterparts, while state-of-the-art multimodal models drop nearly 30 points on reasoning-intensive tasks.Source: §6 · p. 10

CODE OF ETHICS AND ETHICS STATEMENT

• MRMR uses publicly available materials for academic research only, not commercial use.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 10
• The authors state that the dataset contains no private information or harmful content, and that annotators were instructed to avoid websites prohibiting copying or redistribution.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 10
• Most documents come from sources described as free of copyright restrictions, and the underlying datasets carry permissive public licenses.Source: CODE OF ETHICS AND ETHICS STATEMENT · p. 10

REPRODUCIBILITY

• Dataset and evaluation code are publicly available through Hugging Face and GitHub.The paper points to the MRMRbenchmark dataset and MRMR evaluation-code repository.Source: REPRODUCIBILITY · p. 10

A THE USE OF LARGE LANGUAGE MODELS

• LLMs were used only as data-generation tools; no manuscript sections were generated by LLMs.The authors state that they therefore see no plagiarism or scientific-misconduct concerns related to text generation.Source: A THE USE OF LARGE LANGUAGE MODELS · p. 18

B.1 ANNOTATOR BIOGRAPHY

• MRMR involved 24 annotators from Top 500 QS-ranked universities, fluent in English, with independent validators checking relevance labels.The annotator biographies are hidden to protect identity confidentiality.Source: §B.1 · p. 18, Table 4 · p. 18

B.2 ANNOTATION GUIDELINE AND INTERFACE

• The annotation interface separates question understanding, candidate-document evaluation, and creation of a relevant document when candidates are insufficient.For the final step, annotators search for a suitable web page and provide gold evidence in interleaved image–text format.Source: §B.2 · p. 18

B.3 DATA ANNOTATION PAYMENT

• Annotation and validation spanned three months; annotators handled about 50 questions each, with 10 minutes per question and 4 minutes per validation.Pay was 7 USD per hour with an approximately 10% quality adjustment, and annotators were advised to complete no more than 10 questions daily.Source: §B.3 · p. 18

B.4 DATASET CONSTRUCTION PROMPTS

• Dataset-construction prompts are documented in Figures 6–9, with a separate prompt for theorem-content cleaning in Figure 10..Source: §B.4 · p. 19, Figure 10 · p. 23

C.1 THEOREM DATABASE CONSTRUCTION; C.2 WIKIPEDIA CONTENT PROCESSING PIPELINE; C.3 DOCUMENT DEDUPLICATION

• The theorem corpus was embedded and indexed, Wikipedia and scanned PDFs supplied theorem content, and near-duplicate statements with cosine similarity ≥0.85 were collapsed.The corpus retains theorem IDs and original text; processing used MediaWiki Search, MonkeyOCR, and GPT-5 for extraction and cleaning.Source: §C.3 · p. 19, §C.1 · p. 19, §C.2 · p. 19

D.1 NEGATION

• The Negation task uses 200 COCO images to create three positive descriptions and one negative description, with manual review for human-identifiable contradictions.The construction does not involve LLM prompting.Source: §D.1 · p. 20

D.2 VEHICLE DESIGN

• Vehicle-design queries modify DesignQA cases, while corpus rules are segmented from the Formula SAE Rulebook into 700 files organized by rule ID.Queries are reviewed to ensure they depict incorrect designs.Source: §D.2 · p. 20

D.3 TRAFFIC CASE

• Traffic cases use stories and 12 generated images per story, paired with interleaved objectives and corpus files from Singapore driving handbooks.Generated images are manually reviewed and refined; excess image details prompt inclusion of additional corpus IDs in queries.Source: §D.3 · pp. 20–21

E.1 MODELS AND INSTRUCTIONS

• Text retrievers are evaluated by replacing images with LLM-generated captions produced from the standardized prompt “Describe the image.”The captioning model is Qwen2-VL-2B-Instruct.Source: §E.1 · p. 23

E.2 IMPLEMENTATIONS AND MACHINES

• The evaluation follows MTEB conventions with modifications for mixed-modality inputs and uses NVIDIA A100, A6000, or H100 GPUs.A complete open-source dense-model run can finish within four hours on one A100 GPU.Source: §E.2 · p. 23

E.3 DETAILED RESULTS

• Table 7 presents detailed retrieval-model performance for the Knowledge task..Source: Table 7 · p. 24

F.1 FALSE POSITIVE AND FALSE NEGATIVE ANALYSIS

• Cross-query aggregation or sampling from external corpora can still produce false positives and false negatives despite human validation; some queries share relevant documents.Contradiction tasks received no additional human evaluation because they were relatively small and predominantly manually constructed.Source: §F.1 · p. 25
• Among 120 sampled documents, Knowledge had 0 false positives and a 2.5% false-negative rate; Theorem’s combined error was approximately 5.8% (3.3% false negatives, 2.5% false positives).The authors conclude that label noise is insignificant; Table 8 reports the audit.Source: Table 8 · p. 25, §F.1 · p. 25

F.2 ERROR CASE STUDIES

• Engineering and geometry retrieval can match coarse shapes or keywords while missing task-specific geometric conditions, as in a bearing-angle query about a pentagon.The retrieved negative document concerns inscribing a circle in a regular pentagon.Source: §F.2 · p. 25
• A beam query about tributary areas can retrieve static-equilibrium theory instead of the procedural knowledge needed for area calculation.The analysis identifies this as a conceptual mismatch despite the shared engineering and beam context.Source: §F.2 · p. 26
• A P−V graph can prompt retrieval of an isothermal-process document even when the query requires using PV = nRT to identify the highest temperature.The analysis attributes the plausible but incorrect match to recognizing the graph type without deducing its specific physical implication.Source: §F.2 · p. 26
• For a histogram question asking for a percentage, the model may recognize the chart type but fail to extract the bin counts and ranges needed for calculation.The retrieved negative document discusses generic statistical methods.Source: §F.2 · p. 26

F.3 TEST-TIME SCALING IN RETRIEVAL

• Query expansion generally helps weak retrievers, but weaker-VLM expansions degrade stronger retrievers, making expansion quality critical for stronger models.The experiments use Qwen2-VL-2B and Qwen2.5-VL-72B with GME-Qwen2-VL and Ops-MM-Embedding.Source: §F.3 · p. 29
• 55.9 for medical queries and 31.8 for math queries are the best retriever’s scores with strong query expansion, which improves both strong and weak retrievers but remains far from perfect.The results use Qwen2.5-VL-72B query expansion.Source: §F.3 · p. 29

G DATA EXAMPLES

• Examples include music, clinic medicine, biology, and psychology.Source: Figure 13 · p. 30, Figure 14 · p. 31, Figure 15 · p. 32
• Examples also include math, physics, engineering, and business.Source: Figure 17 · p. 34, Figure 18 · p. 35, Figure 19 · p. 36
• Further examples cover negation, vehicle design, and traffic cases.Source: Figure 21 · p. 38, Figure 22 · p. 38, Figure 23 · p. 39