MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

Abdelrahman Abdallah; Mohamed Darwish Mounis; Mahmoud Abdalla; Mahmoud SalahEldin Kasem; Mostafa Farouk Senussi; Mohamed Mahmoud; Mohammed Ali; Adam Jatowt; Hyun-Soo Kang

Source-linked AI summary

MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, Hyun-Soo Kang

arXiv:2601.09562v2Jan 14, 2026cs.IR

TL;DR

Existing retrieval benchmarks largely separate text-only reasoning from multimodal retrieval, despite real-world queries containing images that carry essential information. MM-BRIGHT introduces a four-task benchmark of multimodal, reasoning-intensive retrieval across technical domains and finds that current models struggle across settings, with images sometimes degrading performance.

Problem
Existing benchmarks largely evaluate either text-only reasoning or surface-level multimodal correspondence, despite real-world queries containing images essential to the information need.
Method
MM-BRIGHT evaluates 2,803 StackExchange queries from 29 technical domains through four retrieval tasks spanning text-only to multimodal-to-multimodal retrieval.
Results
Current retrievers struggle across all settings: BM25 reaches 8.5 nDCG@10 in Task 1, while Nomic-Vision reaches 27.6 nDCG@10 in Task 2, below DiVeR's 32.2.
Takeaways & Limitations
The results highlight substantial headroom and indicate that future retrieval systems must better integrate visual understanding with multi-step technical reasoning.

Abstract

from arXiv · show

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.

1 Introduction

MM-BRIGHT addresses the gap between multimodal content and reasoning-intensive retrieval by combining multimodal queries, technical domains, and retrieval tasks of increasing complexity. Its evaluation shows that current models struggle across settings, including cases where adding images lowers performance.

Motivation and contribution: MM-BRIGHT combines multimodal queries, reasoning-intensive relevance, and multiple retrieval task variants across 2,803 queries and 29 technical domains.The queries come from StackExchange and are curated so relevant documents require reasoning rather than simple keyword matching.
Benchmark design: The benchmark defines four tasks: text-to-documents, multimodal-to-documents, multimodal-to-images, and multimodal-to-multimodal retrieval.The tasks progressively test visual context use, visual reasoning, similarity assessment, and joint evaluation of text and images.
Evaluation: 18 retrieval models are evaluated, spanning sparse, dense, reasoning-enhanced, and multimodal architectures.The evaluation covers diverse model architectures.
Results: 8.5 nDCG@10 is achieved by BM25 in Task 1, while DiVeR reaches 32.2 nDCG@10.These results establish the difficulty of the text-only reasoning baseline before multimodal complexity is added.
Results: 27.6 nDCG@10 is achieved by Nomic-Vision in Task 2, below the text-only result, while Task 3 reaches 45.6 and Task 4 reaches 28.0.The reported results are Nomic-Vision for Task 2, GME-2B for Task 3, and CLIP for Task 4.

2 Related Work

Prior benchmarks typically assess either text-only reasoning or surface-level multimodal correspondence, leaving multimodal reasoning-intensive relevance insufficiently covered. MM-BRIGHT combines these dimensions across technical domains.

Reasoning-intensive retrieval: Reasoning-intensive benchmarks such as BRIGHT and RAR-b address deeper inference but remain limited to text-only queries and documents.They motivate reasoning-aware retrievers without evaluating multimodal settings.
Multimodal retrieval: Many multimodal benchmarks evaluate surface correspondence rather than reasoning-intensive relevance.Their evaluations often emphasize visual similarity, object-text matching, or cross-modal semantic alignment.
Dataset construction: Its annotation process uses expert verification for relevant documents and mines similar-but-irrelevant negatives from generated queries and entities.Positive documents may be found with Gemini assistance or accepted-answer links, while negatives are mined using GPT-4o-generated searches.
MM-BRIGHT: MM-BRIGHT combines multimodal queries with reasoning-intensive relevance across 29 technical domains.The benchmark differs from datasets targeting document question answering rather than technical reasoning.

3 MM-BRIGHT Dataset

MM-BRIGHT formulates reasoning-intensive multimodal retrieval over expert-authored StackExchange queries, documents, and images. Its dataset spans diverse technical domains, visual content, and verified relevance judgments.

Task formulation: Queries connect to relevant documents through visual understanding and logical inference about technical principles, rather than simple visual similarity or keyword matching.
Task formulation: MM-BRIGHT defines four retrieval tasks spanning text-only, multimodal-to-text, multimodal-to-image, and multimodal document retrieval.The tasks increase in multimodal complexity from Query →Documents to Query+Image →Documents+Images.
Data collection: The benchmark draws on expert technical questions across 29 StackExchange domains, including STEM, computing, social sciences, and applied fields.Posts contain integral visual elements such as diagrams, code screenshots, and scientific figures.
Data construction: Human annotators construct multimodal queries and positive documents from posts, linked pages, passages, and images, while collecting topically related hard negatives.The construction process uses expert-reviewed annotation and retrieval of relevant web evidence.
Image diversity: The dataset contains diverse visual reasoning material, including photos (27.2%), diagrams (17.1%), charts/graphs (16.1%), screenshots (13.9%), and scientific figures (11.6%).Images are classified across eight categories using GPT-4o.
Image essentiality: Among 1,585 queries, 33.0% contain essential images, 58.1% helpful images, and 8.8% redundant images.Essential images are critical for understanding the query, whereas helpful images provide useful but nonessential context.
Dataset quality: The quality audit reports Readability 4.47, Clarity 4.18, Evidence sufficiency 4.10, and Evidence usefulness 3.80 on five-point scales.The lower usefulness score reflects that many queries require multi-step reasoning rather than direct lexical overlap.

4 Experiments

Experiments show that MM-BRIGHT challenges current retrievers across text-only and multimodal tasks. Reasoning-enhanced models lead text retrieval, but multimodal systems remain inconsistent and often fail to align text with images.

Task 1: Text-only retrieval: BM25 achieves 8.5 nDCG@10 on Task 1, while DiVeR reaches 32.2 nDCG@10, remaining below the 50+ scores typically reported on BEIR.Dense retrievers such as E5 and SFR score 25.3 and 26.9, respectively.
Task 2: Multimodal-to-text retrieval: Multimodal models perform worse than text-only retrieval on Task 2: Nomic-Vision scores 27.6 nDCG@10 versus DiVeR's 32.2.BGE-VL scores 10.0, while Jina-CLIP and GME-7B reach 23.0 and 22.0.
Tasks 3–4: Visual and joint retrieval: GME-2B achieves 45.6 nDCG@10 on Task 3 image retrieval, whereas CLIP reaches 28.0 on the harder Task 4 multimodal document task.Task 4 uses a distinct graded relevance scale, so its scores are not directly comparable with Task 3.
Tasks 3–4: Visual and joint retrieval: Task 4 models commonly retrieve one modality without aligning the complete text-image pair into a unified evidence block.This indicates a modality-alignment failure in multimodal document retrieval.
Cross-task consistency: Model strengths are task-specific: Nomic-Vision leads Task 2 at 27.6, GME-2B leads Task 3 at 45.6, and CLIP never achieves the top score.The results indicate narrow specializations rather than general multimodal reasoning ability.

5 Additional Analysis

Additional analyses examine how captions, image essentiality, query reformulation, and retrieval quality affect performance. They show that visual reasoning remains difficult and that better retrieval is still needed for downstream QA.

Image captions: Adding image captions improves E5 by 7.4 nDCG@10 but reduces DiVeR by 12.0 points.Captions provide useful semantic information for dense retrieval while disrupting the reasoning-based retriever.
Image essentiality: Multimodal models average 20.4 nDCG@10 on helpful-image queries but only 15.0 on essential-image queries.Essential-image performance is also below redundant-image performance at 16.7, and 33.0% of queries contain essential images.
Query reformulation: GPT-4o query reformulation raises E5 from 25.3 to 28.3 but changes DiVeR from 32.2 to 31.8.Reformulation improves some semantic retrievers but offers little benefit for the strongest reasoning-focused retriever.
End-to-end QA: Oracle evidence yields 67.17 end-to-end QA performance, while the best retrieval-based setting trails it by 6.46 points.The result shows a substantial gap between ideal evidence and retrieved evidence in the RAG pipeline.

6 Conclusion

MM-BRIGHT benchmarks reasoning-intensive retrieval across 29 technical domains and four retrieval settings. Evaluation shows current retrievers struggle broadly, with images often degrading performance when visual information is critical.

MM-BRIGHT covers 29 technical domains and four tasks ranging from text-only retrieval to multimodal document retrieval.
Evaluation of 18 models shows that current retrievers struggle across all settings.
Adding images often degrades performance when visual information is critical.
Future retrieval systems must better integrate visual understanding with multi-step technical reasoning.

A.1 Dataset Task 2 Statistics

Dataset 2 supports multimodal retrieval with 1,218 queries and 7,621 annotated positive images across 29 domains. Its construction combines expert-reviewed StackExchange posts, integral images, reasoning-focused positives, and challenging hard negatives.

1,218 queries and 7,621 annotated positive images support Tasks 2–4 across 29 domains.
Selected posts require an accepted or highly voted answer, at least one integral image, and technical complexity beyond direct keyword matching.
Positive documents must provide concepts, technical material, underlying principles, or reasoning steps that help derive an answer rather than merely share keywords.
Hard negatives are topically related and semantically similar but omit the specific reasoning or concepts needed to answer the query.
Reviewers retain only annotations receiving unanimous approval, with disagreements resolved by consensus or removal.

B Caption-based Query Augmentation Results (Task 1)

Caption augmentation helps semantic retrievers but disrupts reasoning-enhanced retrievers, while larger caption models provide diminishing returns. Domain difficulty remains stable across caption models, and captioning cannot replace native multimodal reasoning.

+5.7 to +7.4 points for E5 show consistent gains from captions across caption models.
-12.0 to -12.4 points for DiVeR show severe degradation with all caption models.
32-33 nDCG@10 for E5 across Qwen-3B through Qwen-72B indicates minimal gains from 24× parameter scaling.
14.1 for Quantum Computing and 63.7 for Law mark persistent domain difficulty differences across caption models.
Caption-based approaches cannot replace true multimodal reasoning because they rely on intermediate text generation.

C.1 Image type taxonomy and classifier setup

MM-BRIGHT classifies query images into eight categories to capture diverse visual reasoning demands across technical domains. GPT-4o also labels image essentiality using criteria distinguishing indispensable visual evidence from supplementary or redundant content.

Domain distributions: Image distributions vary by domain, with photos, diagrams, charts, screenshots, mathematical notation, and scientific figures each dominating different technical areas.
Domain distributions: This visual diversity prevents evaluation from being reduced to models specialized in a single image type.
Image type taxonomy: GPT-4o classifies images into eight categories, including diagrams, screenshots, charts, photographs, mathematical notation, scientific figures, mixed content, and other types.
Image essentiality: Essential images contain information that cannot be adequately expressed in text, including configurations, equations, scientific features, error states, and chart patterns.
Image essentiality: GPT-4o achieved 99.8% high-confidence classifications, while domain experts agreed with 94% of judgments on 100 sampled cases.

D.3 Domain-Level Analysis

MM-BRIGHT’s domain analysis shows that image essentiality varies substantially across technical fields, while current multimodal retrievers generally struggle most when images are indispensable. The benchmark therefore targets visual reasoning beyond surface similarity.

Domain patterns: 57.8% of Quantum Computing queries contain essential images, while STEM domains broadly rely on diagrams, spectrograms, cellular structures, molecular diagrams, equations, and sequence visualizations.Quantum Computing, Bioacoustics, Biology, and Chemistry have the highest essential-image rates.
Domain patterns: Philosophy, Economics, and Christianity predominantly use helpful images that illustrate concepts or provide examples rather than irreplaceable visual information.Psychology is an exception, with scientific figures requiring visual analysis.
Image essentiality: Only 9.5% of images are redundant, indicating that Stack Exchange posts usually contain meaningful visual information rather than decorative content.Law, Aviation, and Sustainability have the highest redundant-image rates.
Implications: Caption-based approaches cannot reliably represent technical visual details such as quantum gate configurations or cellular features.The benchmark’s image diversity and essentiality distribution motivate domain-specific visual understanding and evaluation by image importance.
Image essentiality: Essential images require understanding diagrams, scientific principles, or screenshot errors, exceeding simple visual similarity or object detection.The essentiality classifier distinguishes images that are required, useful but supplementary, or unnecessary based on whether text alone identifies the information need.

E.2 Per-Domain Results

Query reformulation helps some semantic retrievers but has mixed, retriever-dependent effects and does not improve reliably with model scale. Larger reformulators can recover performance, yet domain and architecture remain decisive.

Evaluation setup: The experiments evaluate seven vision-language reformulators and ten retrieval models across all 29 domains.The reformulator analyzes query text and associated images to generate an expanded statement of the information need.
Per-model results: GPT-4o reformulation raises E5 from 25.3 to 28.3 nDCG@10 and Rader from 24.9 to 25.2 on average across domains.The same reformulation causes DiVeR to decline from 32.2 to 31.8.
Per-model results: Llama-3.2-11B reformulation lowers BM25 to 8.4 and DiVeR to 31.0, showing that lower-quality reformulations can harm reasoning-enhanced retrieval.E5 changes minimally, reaching 25.6.
Per-model results: Llama-3.2-90B recovers performance, with E5 reaching 27.7, DiVeR 32.0, and Rader 31.6.Rader approaches its GPT-4o reformulation performance.
Scale effects: Qwen2.5-VL scale does not monotonically improve reformulation quality, with the 32B model achieving the highest DiVeR score of 32.7 while smaller variants often match original-query performance.The results suggest a capability threshold for effective reformulation rather than continuous gains from scale.
Cross-model patterns: Semantic retrievers consistently benefit more from reformulation than reasoning-enhanced retrievers, whose results are mixed across models and domains.Technical domains such as Quantum Computing and Cryptography show minimal improvement, whereas Gaming and Law often benefit more.

F LLM-based Dataset Quality Assessment

The dataset-quality audit uses GPT-4o to judge query writing and evidence validity on four 1–5 Likert dimensions. MM-BRIGHT is generally clear and sufficient, but evidence usefulness is comparatively lower.

Assessment procedure: GPT-4o rates each query-positive-passage-answer triple on Readability, Clarity, Evidence usefulness, and Evidence sufficiency using 1–5 Likert scores.The audit targets dataset clarity and evidence validity rather than retrieval-model performance.
Assessment dimensions: Evidence usefulness measures whether passages support reasoning toward the answer beyond topical overlap.It considers mechanisms, procedures, explanations, derivations, examples, technical credibility, and alignment with query intent.
Assessment dimensions: Evidence sufficiency measures whether passages contain enough information to answer correctly, including critical definitions, constraints, parameters, steps, and actionable detail.Evidence can be useful yet incomplete, so usefulness and sufficiency are assessed separately.
Assessment goal: The judge evaluates whether queries are unambiguous and whether annotated positives are relevant for reasoning and sufficient for answering.Answer correctness is separately scored from 0 to 100 by comparing generated answers with reference answers.

H Dataset Examples

The dataset examples span technical and nontechnical domains, illustrating the varied source content and reasoning contexts represented in MM-BRIGHT. Examples include scientific explanations, Earth science, finance, philosophy, computing, and applied topics.

Domain coverage: The examples include Bioinformatics, Crypto, Earthscience, Medicalsciences, Quant, Quantumcomputing, Sustainability, Philosophy, Physics, Robotics, Economics, Biology, Academia, Pm, Gaming, and Travel.These labeled examples demonstrate the benchmark’s breadth of technical and applied domains.
Earth science: The Earth science example defines seismic tomography as imaging Earth’s subsurface with seismic waves and interpreting resulting velocity models.The technique compares waves recorded at different locations and supports analyses from whole-Earth structure to shallow subsurface features.
Example content: One negative example states that earthquake energy is a function of magnitude, illustrating a concise background statement in the example collection.The passage also relates magnitude and seismic moment to radiated earthquake energy.
Domain coverage: The collection also includes examples from social and applied contexts, including Philosophy, Sustainability, Academia, Project Management, Gaming, and Travel.The source examples therefore extend beyond STEM and computing topics.