PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, Hyun-Soo Kang

arXiv:2601.09562v2cs.IR

TL;DR

Existing retrieval benchmarks largely miss queries whose visual elements require reasoning beyond surface-level matching. MM-BRIGHT introduces a 2,803-query benchmark across 29 technical domains and four retrieval tasks, finding that current retrievers struggle across settings, with Nomic-Vision reaching only 27.6 nDCG@10 versus DiVeR’s 32.2 on multimodal-to-text retrieval.

  • Problem

    Existing benchmarks are predominantly text-centric or evaluate multimodal retrieval through surface-level correspondence, leaving multimodal reasoning-intensive retrieval insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    MM-BRIGHT provides 2,803 expert-curated queries across 29 technical domains and evaluates four retrieval tasks spanning text-only to multimodal document retrieval.

    Source: §1 · p. 2

  • Results

    Current retrievers struggle across all tasks; Nomic-Vision scores 27.6 nDCG@10 on multimodal-to-text retrieval, below DiVeR’s 32.2.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    MM-BRIGHT highlights substantial headroom for retrieval systems integrating visual understanding with multi-step technical reasoning.

    Source: §6 · p. 9

Abstract

from arXiv · show

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.

1 Introduction

MM-BRIGHT addresses the gap between multimodal retrieval and reasoning-intensive retrieval with a benchmark combining multimodal queries, technical-domain reasoning, and four increasingly complex retrieval tasks. Evaluation across 18 models shows that current retrievers struggle, with images not consistently improving text retrieval and multimodal document retrieval remaining difficult.

Motivation

  • Images such as diagrams, charts, screenshots, and scientific figures can carry essential information that text alone cannot adequately capture in real-world information needs.Examples include software troubleshooting with screenshots and biological searches involving microscopy images.Source: §1 · p. 1
  • Existing multimodal benchmarks mainly test surface-level visual or textual correspondence, while reasoning-intensive benchmarks such as BRIGHT (Su et al., 2024) and RAR-b (Xiao et al., 2024) remain text-only.MM-BRIGHT targets the resulting gap by requiring multimodal understanding and intensive reasoning simultaneously.Source: §1 · pp. 1–2

Benchmark Contribution

  • MM-BRIGHT contains 2,803 expert-curated real-world queries spanning 29 technical domains, with relevant documents requiring reasoning beyond simple keyword matching.The queries are sourced from StackExchange and cover software engineering, STEM, social sciences, and applied domains.Source: §1 · p. 2

Task Design

  • The benchmark defines four tasks: text-to-document, multimodal-to-document, multimodal-to-image, and multimodal-to-multimodal retrieval, ordered by increasing multimodal complexity.The final task requires jointly evaluating text and images in retrieved documents, whereas the first isolates reasoning intensity without multimodal complexity.Source: Figure 1 · p. 1, §1 · p. 2

Evaluation Results

  • 32.2 nDCG@10 is the best Task 1 score from DiVeR, while BM25 reaches only 8.5; Nomic-Vision scores 27.6 in Task 2, below the text-only baseline, and Task 4 falls to 28.0 with CLIP.Task 3 is stronger at 45.6 with GME-2B, but performance varies widely across models and domains, indicating substantial headroom.Source: §1 · p. 2

2 Related Work

Prior retrieval benchmarks address reasoning mainly in text-only settings, while multimodal benchmarks largely emphasize semantic alignment or document QA rather than reasoning-intensive relevance. MM-BRIGHT fills this gap with multimodal queries and documents spanning 29 technical domains.

Text-only reasoning benchmarks

Multimodal retrieval benchmarks

  • Multimodal retrieval benchmarks cover cross-modal semantic alignment and broader modality combinations, but mostly evaluate surface correspondence rather than reasoning-intensive relevance.Expert-content datasets may address document QA, but not technical reasoning (Macé et al., 2025).Source: §2 · pp. 2–3

MM-BRIGHT annotation

  • MM-BRIGHT’s annotation process uses AI-assisted or accepted-answer discovery, manual relevance verification, and GPT-4o-generated searches to mine similar but irrelevant negatives.Queries are multimodal Stack Exchange posts, and documents include sources such as Wikipedia, blogs, research papers, and technical documentation.Source: Figure 2 · p. 3

MM-BRIGHT

  • MM-BRIGHT combines multimodal queries with reasoning-intensive relevance across 29 technical domains, addressing limitations in prior text-only, alignment-focused, and document-QA benchmarks.The benchmark comparison is summarized in Table 1.Source: §2 · pp. 2–3

3 MM-BRIGHT Dataset

MM-BRIGHT is a reasoning-intensive multimodal retrieval benchmark built from diverse technical StackExchange queries, evaluated across four retrieval tasks and supported by verified document and image annotations. Its images are broadly diverse, usually useful or essential, and the dataset receives high readability, clarity, and evidence-sufficiency scores.

3.1 Task Formulation

  • MM-BRIGHT evaluates four tasks: Query →Documents, Query+Image →Documents, Query+Image →Images, and Query+Image →Documents+Images.Relevant documents require reasoning traces involving visual understanding and logical inference rather than simple visual similarity or keyword matching.Source: §3.1 · p. 3

3.2 StackExchange Multimodal Queries

  • The benchmark draws on 29 technical StackExchange domains spanning STEM, computing, social sciences, and applied fields, whose posts integrate diagrams, screenshots, and scientific figures.Query-document pairs use user posts as queries and documents referenced in answers as sources.Source: §3.2 · p. 3
  • 7,621 verified images across 1,218 queries support multimodal retrieval after GPT-4o classification and unanimous approval by domain-knowledgeable human experts.The corpus includes positive and hard-negative web pages, with negatives selected to be topically related without satisfying query requirements.Source: §3.2 · p. 4

3.3 Image Type Diversity

  • Photos comprise 27.2% of query images, followed by diagrams at 17.1%, charts/graphs at 16.1%, screenshots at 13.9%, and scientific figures at 11.6%.The eight-category distribution varies by domain, including charts/graphs in Economics at 83.0% and screenshots in Ask Ubuntu at 95.1%.Source: §3.3 · p. 4

3.4 Image Essentiality Analysis

  • 90.0% of queries contain essential or helpful images, comprising 33.0% essential, 58.1% helpful, and 8.8% redundant images.Essential images are critical for understanding, helpful images provide useful context, and redundant images add no retrieval value.Source: §3.4 · p. 5

3.5 Dataset Quality Assessment

  • The dataset scores 4.47 for readability, 4.18 for clarity, 4.10 for evidence sufficiency, and 3.80 for evidence usefulness on 1–5 Likert criteria.The lower usefulness score reflects that many queries require multi-step reasoning rather than direct lexical overlap.Source: §3.5 · p. 5

4 Experiments

Experiments evaluate 18 text-only and multimodal retrieval models across MM-BRIGHT’s four tasks, showing that reasoning-intensive multimodal retrieval remains difficult and current models lack robust visual reasoning. Performance depends strongly on the retrieval objective, with visual similarity easier than unified text-image evidence retrieval.

4.1 Experimental Setup

  • The evaluation covers 18 representative retrieval models, including sparse, dense, reasoning-enhanced, proprietary, contrastive vision-language, and multimodal embedding models.Tasks 1–3 use binary relevance, while Task 4 assigns rel=2 to complete gold passage-image pairs, rel=1 to passages without images, and rel=0 to incorrect passages.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 8.5 nDCG@10 for BM25 and 32.2 for DiVeR on Task 1, while Nomic-Vision reaches only 27.6 on Task 2, below the best text-only model.Task 1 scores remain far below the 50+ nDCG@10 typical of BEIR; adding images does not improve retrieval, and BGE-VL reaches only 10.0, matching BM25.Source: Table 3 · p. 6, Table 4 · p. 6, §4.2 · p. 6
  • 45.6 nDCG@10 is achieved by GME-2B on Task 3, whereas Task 4 reaches only 28.0 for CLIP and uses graded relevance, preventing direct comparison with Task 3.Models often retrieve one modality without aligning the complete text-image pair into unified evidence.Source: §4.2 · pp. 6–7, Table 5 · p. 7
  • Nomic-Vision leads Task 2 at 27.6 but scores 27.1 on Task 3, while GME-2B reverses the pattern with 19.5 and 45.6, respectively.CLIP is balanced but never best, scoring 10.4 on Task 2, 33.6 on Task 3, and 28.0 on Task 4; BGE-VL scores 10.0 on both Tasks 2 and 4.Source: §4.2 · p. 7

5 Additional Analysis

Additional analyses show that captioning and query reformulation provide inconsistent or limited retrieval benefits, while multimodal retrievers struggle most when visual evidence is essential. Retrieval improves end-to-end QA, but the best system remains 6.46 points below oracle evidence.

Image captioning

  • +7.4 nDCG@10 lifts E5 with image captions, but DiVeR drops 12.0 points, showing that captions can help semantic retrieval while disrupting reasoning-based retrieval.E5 rises from 25.3 to 32.7, whereas DiVeR falls from 32.2 to 20.2; BM25 improves from 8.5 to 9.8 with GPT-4o captions, while ReasonIR remains around 29–31.Source: §5 · p. 7, Figure 5 · p. 8

Image essentiality

  • 20.4 average nDCG@10 on helpful-image queries falls to 15.0 when images are essential, below 16.7 for redundant images.Nomic-Vision shows a 29.1-to-25.2 decline, and essential visual information is indispensable for 33.0% of queries.Source: §5 · pp. 7–8, Figure 6 · p. 8

Query reformulation

  • 28.3 nDCG@10 is reached by E5 after GPT-4o reformulation, but DiVeR slightly declines from 32.2 to 31.8.Rader gains 2.8 points, BM25 improves from 8.5 to 9.8 with GPT-4o but can degrade with smaller models, and larger reformulators do not consistently outperform smaller ones.Source: Figure 7 · p. 8, §5 · p. 8

End-to-end QA

  • 67.17 QA score is achieved with oracle evidence, while the best retrieved setting trails oracle by 6.46 points.The end-to-end RAG evaluation uses Llama-3.2-90B-VL and GPT-4 judging on a 0–100 scale.Source: Table 7 · p. 8, §5 · pp. 8–9

6 Conclusion · Appendix Contents · A Dataset Construction and Annotation Protocol

MM-BRIGHT is a 29-domain benchmark spanning four retrieval tasks, and evaluation of 18 models shows substantial difficulty across settings, especially when visual information is critical.

6 Conclusion

  • Current retrievers struggle across all four MM-BRIGHT retrieval tasks spanning 29 technical domains.The benchmark ranges from text-only retrieval to multimodal document retrieval.Source: §6 · p. 9
  • Adding images often degrades performance when visual information is critical, highlighting the need to integrate visual understanding with multi-step technical reasoning.These results indicate substantial headroom for future retrieval systems.Source: §6 · p. 9

A.1 Dataset Task 2 Statistics … A.7 Prompt templates

This section details the construction and annotation of Dataset 2 for multimodal retrieval, including query selection, multimodal construction, reasoning-focused positives, challenging negatives, review procedures, and domain-specific guidance. It also covers domain-aware annotation notes, GPT-4o-assisted hard-negative mining, and image-relevance labeling.

A.1 Dataset Task 2 Statistics

  • Dataset 2 contains 1,218 queries and 7,621 annotated positive images across 29 domains for multimodal Tasks 2–4.It includes only queries whose image annotations were completed for multimodal evaluation.Source: §A.1 · p. 12, Table 8 · p. 12

A.2.1 Post Selection Criteria

  • Posts are selected for accepted or highly upvoted answers, integral technical images, and questions requiring reasoning beyond simple keyword matching.Eligible images include diagrams, charts, screenshots, or scientific figures that are essential, visible, and non-corrupted.Source: §A.2.1 · p. 12

A.5 Quality control and review

  • The annotation framework excludes decorative or corrupted images, subjective or speculative posts, duplicate questions, answer-only documents, and unrelated negatives.It also favors authoritative sources and passages with enough context to be independently understandable.Source: §A.2.1 · p. 12, §A.5 · p. 14

A.2.2 Constructing Queries

  • Multimodal queries combine cleaned titles and bodies with preserved technical content and all integral body images in their original order.Images must be high quality and clearly visible, with paths or URLs recorded for reference.Source: §A.2.2 · p. 12

A.3 Positive document construction

  • Positive documents must provide concepts, technical material, principles, or reasoning steps that materially help derive an answer rather than merely sharing keywords.Annotators discover candidates through answer-linked pages and Gemini-assisted web search, then extract coherent, self-contained relevant passages and useful images.Source: §A.3 · p. 13

A.4 Hard negative mining

  • Hard negatives are topically related but omit the specific reasoning or technical details needed to answer the query, preventing reliance on simple semantic matching.The procedure generates search queries with GPT-4o, collects exactly 20 URLs per query, and extracts challenging related passages.Source: §A.4 · pp. 13–14

A.5 Quality control and review

  • Quality control checks image necessity, answer quality, positive-document reasoning support, hard-negative difficulty, image visibility, and exactly 20 hard-negative URLs per query.Two PhD students or domain experts review annotations, retaining only unanimously approved examples and resolving disagreements by consensus or discarding them.Source: §A.5 · p. 14

A.6 Domain-specific annotation notes

  • Domain guidance prioritizes authoritative sources and preserves domain-relevant evidence, including equations and figures for STEM, documentation and error screenshots for computing, and interpretable charts for social sciences.Applied domains balance technical documentation with practical guides, while hard negatives remain related but miss the specific application or problem.Source: §A.6 · pp. 14–15
  • Domain-specific notes distinguish visual elements showing specific scenarios or configurations from hard negatives discussing the same domain but different specific cases.These notes clarify why topical similarity alone does not establish relevance.Source: §A.6 · p. 15

A.7.1 Hard Negative Mining Prompt

  • GPT-4o generated queries for topically similar but technically irrelevant content, and annotators collected 20 hard-negative URLs per query from Google.The generated analysis also included entities and events to help construct effective negative search queries.Source: §A.7.1 · p. 15

A.7.2 Image relevance annotation prompt

  • 7,621 images across 1,218 Dataset 2 queries were annotated by GPT-4o, which classified each query-image pair with a JSON label and rationale.The annotation evaluates each image against the query, positive passages, and ground-truth answer.Source: §A.7.2 · pp. 15, 17
  • Images were labeled positive when they illustrated query-relevant concepts or supporting diagrams, charts, visual evidence, figures, tables, screenshots, code, or UI elements.Images were labeled negative when unrelated, decorative, tangential, corrupted, illegible, or depicting different topics or domains.Source: §A.7.2 · p. 17

B Caption-based Query Augmentation Results (Task 1) · C Image Type Breakdown by Domain

Caption augmentation helps semantic retrievers but substantially disrupts reasoning-enhanced retrievers, with gains plateauing as caption models scale. Domain difficulty persists across captioning methods, indicating that captions cannot replace native multimodal reasoning.

B Caption-based Query Augmentation Results (Task 1)

  • +5.7 to +7.4 nDCG@10 points improve E5 with captions, whereas DiVeR drops by 12.0 to 12.4 points across caption models.These contrasting responses indicate that semantic retrievers exploit textual descriptions, while reasoning-enhanced retrievers are sensitive to verbose generated input.Source: B Caption-based Query Augmentation… · p. 18
  • 14.1 nDCG@10 is the best Quantum Computing result with GPT-4o captions, whereas Law reaches 63.7 with Qwen-7B captions.Relative domain difficulty remains consistent across caption models, with Quantum Computing challenging and Law easier.Source: B Caption-based Query Augmentation… · p. 18
  • Caption-based augmentation offers a simple way to add visual information to text-only retrievers but cannot replace retrieval architectures that natively process visual and textual signals.The reported retriever-specific gains and degradations motivate multimodal systems that avoid relying on intermediate text generation.Source: B Caption-based Query Augmentation… · p. 18
  • 32.0 nDCG@10 for E5 with Llama-3.2-90B captions exceeds its 31.0 result with Llama-3.2-11B captions, but reasoning retrievers remain below no-caption baselines.Caption quality can improve semantic retrieval, yet it does not resolve the mismatch between generated descriptions and reasoning-based relevance assessment.Source: Table 10 · p. 20
  • 32.3 nDCG@10 for E5 with Qwen-2.5-3B captions is comparable to larger caption models, while DiVeR remains at 20.2.The degradation pattern persists across Qwen model sizes, suggesting independence from caption-model scale and architecture.Source: Table 11 · p. 21
  • 32.7 nDCG@10 is E5’s best Qwen-caption result with Qwen-2.5-32B, only 0.4 points above Qwen-2.5-7B despite substantially greater scale.Reasoning-enhanced retrievers remain around 20–21 nDCG@10 across Qwen sizes, showing diminishing returns from larger captions.Source: Table 13 · p. 23
  • 31.8 nDCG@10 for E5 with GPT-4o captions trails the 32.7 achieved with Qwen captions, while BM25 reaches its caption-augmented high of 9.8.Extremely detailed captions may add irrelevant information for E5, whereas GPT-4o’s lexical diversity benefits BM25; DiVeR reaches only 21.2.Source: Table 15 · p. 25

C.1 Image type taxonomy and classifier setup … D.2 Prompt template

MM-BRIGHT spans diverse visual content across 29 domains, requiring models to reason over multiple image types and their relationships to textual queries. Its image essentiality analysis uses structured GPT-4o classifications to distinguish critical, supplementary, and unnecessary visual information.

C.1 Image type taxonomy and classifier setup

  • MM-BRIGHT’s domains exhibit distinct image-type distributions, so retrieval requires understanding diverse visual representations rather than specializing in one content type.Examples include photo-dominant, diagram-dominant, chart-dominant, screenshot-dominant, mathematical, and scientific-figure domains.Source: §C.1 · p. 26

C.2 Per-domain distributions (tables/figures)

  • GPT-4o classified 1,585 query images into eight image types to analyze visual reasoning challenges across MM-BRIGHT.The classification categories cover diagrams, screenshots, data visualizations, photographs, mathematical notation, scientific figures, mixed content, and other images.Source: §C.2 · p. 26, Figure 12 · p. 31
  • 83.0% of Economics images are charts, 95.1% of Ask Ubuntu images are screenshots, and 75.2% of Aviation images are photographs, illustrating strong domain-specific distributions.Other examples include 61.1% diagrams in Quantum Computing and 66.9% scientific figures in Chemistry.Source: Figure 8 · p. 27, Figure 9 · p. 28, Figure 10 · p. 29

D Image Essentiality Analysis

  • Image essentiality is categorized as essential, helpful, or redundant according to whether removing images would prevent understanding the information need or retrieving relevant documents.Essential images include configurations, equations, scientific visual features, error screenshots, and charts whose visual pattern is queried.Source: D Image Essentiality Analysis · p. 32, §D.1 · p. 32

D.1 Essentiality labeling method

  • The essentiality criteria treat decorative, duplicated, tangential, and generic stock images as redundant, while illustrations, reference images, explanatory diagrams, and partial screenshots are helpful.Helpful images add context when the query text already suffices; redundant images add no retrieval information.Source: §D.1 · p. 32
  • 99.8% of 1,218 essentiality classifications had high confidence, while expert verification of 100 samples reached 94% agreement with GPT-4o.Disagreements mainly involved borderline helpful-versus-essential cases rather than clear misclassifications.Source: §D.1 · p. 32

D.2 Prompt template

  • The GPT-4o essentiality prompt uses detailed examples, decision criteria, and a structured output format to produce consistent and explainable classifications across domains.Its design helps distinguish necessary images from supplementary or decorative ones.Source: §D.2 · p. 32

D.3 Domain-Level Analysis … E.1 Reformulation Prompt

MM-BRIGHT images are often essential to technical retrieval, with strong variation across domains and consistent performance degradation when visual information is critical. These findings motivate domain-sensitive visual reasoning and query reformulation that analyzes both text and images before retrieval.

D.3 Domain-Level Analysis

  • Essential image rates are highest in Quantum Computing (57.8%), Bioacoustics (56.4%), Biology (53.1%), and Chemistry (43.1%), where diagrams, spectrograms, cellular structures, and molecular representations resist precise textual description.Mathematics (42.1%) and Bioinformatics (43.5%) also have high essential rates because of complex equations and sequence visualizations.Source: §D.3 · p. 32
  • Image essentiality varies substantially across domains: technical computing ranges from Quantum Computing (57.8%) and Cryptography (31.4%) to Bitcoin (24.1%) and Ubuntu (14.8%), while social sciences generally favor helpful images.Philosophy (76.3%), Economics (67.9%), and Christianity (72.7%) have high helpful-to-essential ratios, whereas Psychology has 26.1% essential images; applied domains also vary by image type.Source: §D.3 · p. 34
  • Only 9.5% of images are redundant overall, with redundancy highest in Law (30.4%), Sustainability (20.8%), and Aviation (14.4%), indicating meaningful visual information rather than decorative content.Essential images are absolutely required to understand the query; redundant images repeat information already in the text or are decorative.Source: §D.3 · p. 34

D.4 Implications

  • The 90.5% combined prevalence of essential and helpful images implies that caption-based retrieval is insufficient and that domain-specific visual reasoning is required.Technical images may encode precise gate configurations or cellular features that text cannot adequately capture, and essentiality ranges from 8.7% in Law to 57.8% in Quantum Computing.Source: §D.4 · p. 34
  • MM-BRIGHT requires reasoning beyond visual similarity because essential images may require interpreting diagrams, scientific principles, or screenshot errors, and its 31.2% essential-image rate makes essentiality-aware evaluation important.The findings motivate benchmarks that distinguish decorative from essential images rather than treating visual input uniformly.Source: §D.4 · p. 34

D.5 Per-model results by essentiality

  • Current multimodal retrievers show consistent negative performance gaps between essential and redundant images across most models and domains, indicating difficulty when visual information is critical.Tables 17–23 report nDCG@10 by essentiality category and the essential–redundant performance gap for all multimodal models across 29 domains.Source: §D.5 · p. 34, Table 17 · p. 35, Table 18 · p. 35

E Query Reformulation Detailed Results

  • Query reformulation uses vision-language models to analyze each query with its associated images and generate an expanded statement of the underlying information need before retrieval.The reformulation prompt instructs the model to reason about what information would help answer the question, bridging surface-level query text and deeper retrieval needs.Source: E Query Reformulation Detailed Results · p. 38, §E.1 · p. 38

E.2 Per-Domain Results

Across 29 domains, query reformulation helps semantic retrievers most reliably but produces mixed or negative effects for reasoning-enhanced and lexical retrievers. Performance varies by reformulation model, retriever, and domain, with larger models not consistently yielding better results.

Qwen2.5-VL reformulation

  • 32.7 nDCG@10 is achieved by Qwen2.5-VL-32B on DiVeR, while the 72B variant is slightly lower for other retrievers.The 3B and 7B variants perform comparably to original queries for most retrievers, suggesting a capability threshold for effective reformulation.Source: §E.2 · p. 38, Table 29 · p. 44, Table 30 · p. 45

Cross-model observations

  • Semantic retrievers such as E5 and SFR consistently benefit from reformulation, whereas reasoning-enhanced retrievers such as DiVeR and ReasonIR show mixed effects.BM25 is highly sensitive: GPT-4o improves it by +1.3, while smaller reformulation models cause degradation.Source: §E.2 · p. 38

Cross-domain observations

  • Reformulation gains vary substantially by domain, with minimal improvement in Quantum Computing and Cryptography but larger benefits in Gaming and Law.Larger reformulation models do not consistently outperform smaller ones, indicating diminishing returns beyond a capability threshold.Source: §E.2 · p. 38

GPT-4o reformulation

  • +3.0 nDCG@10 lifts E5 from 25.3 to 28.3 with GPT-4o reformulation, while DiVeR declines from 32.2 to 31.8.GPT-4o also raises Rader from 24.9 to 25.2 on average across domains, but explicit reformulation may interfere with reasoning-enhanced retrieval.Source: §E.2 · p. 38, Table 24 · p. 39

Llama reformulation

  • With queries reformulated by Llama-3.2-90B, E5 reaches 27.7, DiVeR achieves 32.0, and Rader improves to 31.6 in NDCG@10 performance.Source: §E.2 · p. 38, Table 26 · p. 41

F LLM-based Dataset Quality Assessment · F.1 Judge prompt and Metric Definitions · G RAG answer evaluation prompt (GPT-4 judge)

MM-BRIGHT is audited with GPT-4o ratings of query quality and evidence validity across four Likert dimensions, while RAG answers are scored by GPT-4 for reference-answer coverage. The protocols distinguish evidence relevance from completeness and assess answer correctness separately from dataset quality.

F LLM-based Dataset Quality Assessment

  • GPT-4o rates each query–positive passages–answer triple on Readability, Clarity, Evidence usefulness, and Evidence sufficiency using integer 1–5 Likert scores.Scores are aggregated by domain as mean ratings in Table 31.Source: F LLM-based Dataset Quality Assessment · p. 46, §F.1 · p. 46, Table 31 · p. 47
  • Overall scores indicate that queries are generally readable and clear, positive evidence is typically sufficient, and usefulness is comparatively lower.The lower usefulness aligns with MM-BRIGHT’s focus on reasoning-intensive retrieval.Source: Figure 15 · p. 46

F.1 Judge prompt and Metric Definitions

  • The audit targets dataset clarity and evidence validity rather than retrieval-model performance, verifying that queries are well-formed and positives support reasoning beyond keyword overlap.The judge assesses whether annotated positives are relevant for reasoning and sufficient for answering.Source: §F.1 · pp. 46–47
  • Evidence usefulness measures whether positives provide relevant, credible material and reasoning support aligned with the query’s intent.The assessment looks beyond topical overlap for mechanisms, procedures, explanations, derivations, or examples that enable reasoning.Source: §F.1 · p. 46
  • Evidence sufficiency measures whether positives contain enough complete, actionable information to answer correctly, including critical definitions, constraints, parameters, and steps.Low scores indicate major gaps that prevent reliable answers, whereas high scores allow only minor omissions.Source: §F.1 · pp. 46–47
  • Usefulness and sufficiency are distinct: evidence may be relevant but incomplete, or answer-bearing yet weak for reasoning.This distinction reflects the benchmark’s reasoning-focused retrieval objective.Source: §F.1 · p. 47

G RAG answer evaluation prompt (GPT-4 judge)

  • For RAG answer evaluation, a GPT-4 judge receives the query, generated answer, and reference answer, then returns a brief rationale and a 0–100 coverage score.The protocol follows BRIGHT and evaluates how much of the reference answer the generated response covers.Source: G RAG answer evaluation prompt (GPT-4… · p. 49

H Dataset Examples … Example negative document

MM-BRIGHT provides randomly sampled query examples across diverse technical domains, pairing positive and negative documents to illustrate retrieval distinctions. The examples include substantive documents on seismic tomography and Google Scholar profile verification alongside related but mismatched negatives.

H Dataset Examples

  • The dataset examples span bioinformatics, crypto, earth science, medical sciences, quant, and quantum computing domains, with each table presenting one positive and one negative document.Additional examples cover sustainability, philosophy, physics, robotics, economics, biology, academia, PM, gaming, and travel.Source: Table 32 · p. 51, Table 33 · p. 52, Table 34 · p. 53

Example positive document

  • Seismic tomography is presented as a positive document because it explains imaging Earth’s subsurface with seismic waves and interpreting resulting velocity models.The passage describes applications ranging from whole-Earth structure to the upper few meters below the surface.Source: Example positive document · p. 53

H Dataset Examples

  • The tables and document pairs demonstrate that MM-BRIGHT examples require distinguishing closely related content rather than relying only on shared topical vocabulary.The earth-science pair shares earthquake-related terminology, while the Google Scholar pair shares profile-verification terminology but addresses different purposes.Source: Example positive document · pp. 53, 63, Example negative document · pp. 53, 63

Example negative document

  • The paired negative document instead discusses how earthquake magnitude and seismic moment relate to radiated energy.Its earthquake-energy focus differs from the positive document’s subsurface-imaging topic.Source: Example negative document · p. 53

Example positive document

  • Another positive document examines confusion over Google Scholar profile verification and explains that institutional email verification does not establish identity verification.It contrasts Google Scholar’s wording with verification practices for social-media profiles.Source: Example positive document · p. 63

Example negative document

  • The paired negative material gives procedural instructions for adding a Western University affiliation and managing article groups and profile updates in Google Scholar.These instructions concern profile setup rather than the positive document’s investigation of misleading verification implications.Source: Example negative document · p. 63

Related papers

Loading 2601.09562v2…