Source-linked AI summary
MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments
Han Wang, David Wan, Hyunji Lee, Thinh Pham, Mikaela Cankosyan, Weiyuan Chen, Elias Stengel-Eskin, Tu Vu, Mohit Bansal
TL;DR
Real-world search requires agents to reason across heterogeneous, noisy, and conflicting multimodal evidence without being told which modalities matter. MERRIN introduces a human-annotated benchmark for this setting and finds low accuracy even under multimodal search, with agents trailing humans in accuracy and efficiency.
Problem
Existing benchmarks insufficiently evaluate multimodal web search when queries lack modality cues and evidence is noisy, incomplete, or conflicting.
Method
MERRIN is a human-annotated benchmark using natural-language queries, video and audio, and noisy multimodal evidence requiring multi-hop reasoning.
Results
22.3% average accuracy across all runs and 40.1% for the best-performing configuration show that MERRIN is highly challenging.
Takeaways & Limitations
MERRIN provides a testbed for evaluating search agents' multimodal evidence retrieval and reasoning in noisy web environments.
Takeaways & Limitations
The thinking-effort analysis is conducted with GPT-5.4-mini because Gemini models do not support disabling thinking.
Abstract
from arXiv · showhide
Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.
1 Introduction
MERRIN targets multimodal search questions that require agents to infer needed modalities, retrieve evidence, and reason across noisy web sources. It evaluates these capabilities with natural-language queries, diverse modalities, and conflicting or incomplete evidence.
- MERRIN evaluates modality selection, multimodal evidence retrieval, and multi-hop reasoning over noisy, conflicting, and incomplete web sources.Its overview distinguishes reasoning, modality, and retrieval errors as separate failure modes.
- The benchmark uses natural-language queries without explicit modality cues, requiring agents to infer which modalities are necessary.
- Video and audio are included alongside text, images, and tables as evidence modalities.
- Each question induces retrieval of relevant evidence together with incomplete, conflicting, or misleading distractors.
- 22.3% average accuracy across all runs and 40.1% for the strongest configuration show that MERRIN remains highly challenging.
- Humans achieve 71.4% accuracy while using nearly 3× fewer searches and selecting sources more precisely than the best agentic system.The reported source-selection precision is 38.1% for humans versus 1.8% for agents.
2 MERRIN
MERRIN is a human-annotated, expert-vetted benchmark built from questions requiring non-text evidence, unique answers, and multimodal reasoning. Its dataset combines four source types and uses multi-round human quality control.
- MERRIN presents a human-annotated benchmark for retrieving modalities and reasoning over noisy, conflicting multimodal evidence.
- Question Design: Questions contain no explicit modality cues, require non-text evidence, and have exactly one correct, short, and unambiguous answer.
- Question Design: Questions cover multi-hop reasoning and multimodal conflict resolution as their two reasoning classifications.
- Quality Control: Approximately 39.5% of candidate questions were rejected in the first review round, while 45.3% of those were revised and accepted in the second round.
- Human Annotators: Six annotators constructed and reviewed questions, with separate construction and quality-control roles and no self-review.
- Data Statistics: 162 questions span text, image, video, and table sources, with non-text evidence serving as answer sources and reasoning components in comparable proportions.
- Data Statistics: 73.5% of questions require both multi-hop reasoning and multimodal conflict resolution.
3 Experiments
MERRIN evaluates ten-model search-augmented agents across three search settings using accuracy and analyses of multimodal retrieval, multi-step reasoning, and noisy-web behavior. Results show low overall performance, benefits from broader evidence access, and recurring failures from modality bias, error propagation, and over-exploration.
- Overall Performance: 22.3% average accuracy across all runs shows that MERRIN is challenging for search-augmented agents.Across six models evaluated in all settings, accuracy was 17.3% without search, 23.1% with Native Search, and 33.7% with Agentic Multimodal Search.
- Overall Performance: 40.1% is the best overall result, achieved by Gemini-3.1-Pro with Agentic Multimodal Search.The same comparison reports 33.3% as the highest Native Search accuracy, achieved by Gemini Deep Research Agent.
- Bias Toward Text Modality: 87.7% of retrieved evidence is text, compared with 6.8% images and 5.5% video and audio combined.The dataset distribution is more balanced: 31.4% text, 35.9% image, and 28.8% video and audio.
- Error Propagation in Multi-Step Retrieval: 57.7% of incorrect multi-step predictions fail at the first step, versus 42.3% at the second step.The analysis identifies initial evidence identification as a frequent failure point, with early errors propagating to final answers.
- Analysis Across Dataset Axes: 28.0% accuracy occurs when both non-text modalities are required in the reasoning chain and as the answer.Performance is 45.8% when non-text modalities are required in the chain and 45.3% when required as the answer; it is 34.5% when multi-hop and multimodal conflict challenges coincide.
- Over-Exploration in Noisy Web Environments: 33.1% of questions time out for Gemini Deep Research, which may search for up to 15 minutes without producing an answer.Gemini Pro models under Native Search trigger TOO MANY TOOL CALLS on 12.7% of questions, compared with 3.1% for Flash and 0.4% for Lite.
4 Additional Analysis
Additional analyses show that broader modality access improves performance, while search distractors, inefficient source selection, and weak synthesis limit agents relative to humans.
- 4.1 Impact of Adding Video Processing Tool: 5.7% average absolute improvement follows adding video processing to Native Search across four Gemini agents.Native Search otherwise cannot process video or audio during search.
- 4.2 Impact of Thinking Effort: 8.6% is the largest thinking-effort gain, observed in Agentic Multimodal Search; Native Search gains 6.8% and No Search gains 3.1%.The comparison uses GPT-5.4-mini across three search frameworks.
- 4.3 Decomposing the Performance Gap: Search vs. Reasoning: +3.3% from Gold Sources Injection, +2.1% from Gold Sources Only, and +2.2% from Gold Sources Prompting together yield a 7.6% gain, from 40.1% to 47.7%.The results indicate that distractor selection and reasoning remain limiting even when gold evidence is available.
- 4.4 Human Performance: 71.4% human accuracy exceeds Agentic Multimodal Search at 40.1% and Native Search at 30.9%, while humans use fewer resources and achieve 38.1% URL precision versus 1.8%.Humans average 2.9 searches and 2.9 website visits, compared with 9.1 searches and 3.5 visits for Agentic Multimodal Search.
- 4.4 Human Performance: 12.2% is the human gain from five-minute-budget accuracy to overall accuracy, compared with 6.1% for Agentic Multimodal Search and 1.3% for Native Search.The passage attributes the contrast to humans productively leveraging extra time while agents gain little from longer reasoning chains.
- 4.5 Human Error Analysis: 43% of human errors are small counting mistakes, 29% involve the wrong detail from the right source, and 14% are partial or imprecise answers.Only 14% of errors fall into the genuinely incorrect Others category.
5 Related Work
Prior benchmarks often guide agents toward modalities or use synthetically constructed complexity, leaving multimodal reasoning over naturally noisy web evidence underexplored.
- Multimodal Search Benchmarks: Many multimodal search benchmarks provide multimodal inputs or explicit modality cues, limiting evaluation of autonomous modality identification and retrieval.These designs guide agents toward which modalities to retrieve rather than testing independent modality selection.
- Benchmarks for Reasoning under Web Noise: Prior web-noise benchmarks show that ambiguous, conflicting, and incomplete information degrades performance, but multimodal settings often construct complexity synthetically or over predefined evidence.This leaves naturally occurring multimodal web noise less directly represented.
6 Conclusion
MERRIN is a human-annotated benchmark for multimodal evidence retrieval and reasoning over noisy web environments. Its evaluations show that current agents remain substantially less accurate and efficient than humans.
- 6 Conclusion: MERRIN uses natural-language queries without modality cues, includes video and audio, and requires reasoning over noisy, conflicting, and incomplete web evidence.The benchmark evaluates ten LLM-powered agents across three search settings.
- 6 Conclusion: 22.3% average accuracy across all runs and 40.1% for the best configuration demonstrate that MERRIN is highly challenging.Humans search fewer, more precise queries and use more diverse modalities while achieving higher accuracy.
Ethics Statement
The dataset uses publicly available web content that may contain private or sensitive information, so the authors apply human annotation and careful review.
- Ethics Statement: All data is screened to exclude private, biased, or harmful content.The authors describe human annotation and careful review as mitigation measures.
A Limitations
MERRIN’s evaluation may be affected by Google Search ranking biases, limited question diversity, and changing web content that can hinder reproducibility.
- Google Search is the primary engine, so ranking algorithms may introduce benchmark-specific biases.
- The dataset contains 162 questions, which may not capture the full diversity of real-world multimodal queries.
- Dynamic web content may become unavailable or change, potentially affecting reproducibility.
B.1 Data Collection Details
MERRIN constructs human-reviewed, multi-hop questions that require non-text evidence without explicitly naming the necessary modality. The benchmark records answer reasoning, supporting resources, source types, temporal characteristics, and verification outcomes.
- Annotation Fields: Annotators record ground-truth answers, reasoning explanations, supporting URLs, source types, multimodal roles, and reasoning types for each question.
- Question Construction: Questions may combine an existing question–answer pair with additional evidence to create new multi-hop questions, with non-text sources from at least two types encouraged where possible.
- Review Process: Each question undergoes multi-round review for answer correctness, clarity, difficulty, and non-text modality requirements.
- Non-Text Modality Verification: Non-text requirements are verified through standard and adversarial text-only searches, and a question passes only when at least one sub-question remains unresolved by both.
- Review Process: 39.5% of initial candidates were rejected in the first round, and 45.3% of rejected questions were revised and accepted in the second round.
- Core Requirements: MERRIN questions avoid explicit modality cues while requiring at least one reasoning step that cannot be resolved through text-only web search.
C Experiments
Experiments compare models across no-search, native-search, and agentic multimodal-search settings, using model-specific modality and context capabilities. Responses are graded with an LLM-as-judge prompt adapted from BrowseComp.
- Model Configuration: Model comparisons account for maximum context lengths, direct-input modalities, and modalities processed by built-in search.
- Search Settings: Models are evaluated in No Search, Native Search, and Agentic Multimodal Search settings.
- Search Settings: Agentic Multimodal Search adds webpage interpretation and custom tools to extend models’ effective modality coverage.
- Evaluation: Evaluation uses an LLM-as-judge with the same prompt as BrowseComp.
D Human Evaluation
The evaluation materials specify human-evaluation guidance and an automated correctness judgment based on the question, response, and precise correct answer. A table isolates the effect of adding video processing to search.
- Human Evaluation: Human evaluation guidelines are provided separately in Figure 10.
- Automated Evaluation: The autorater judges whether a response is correct using the question, response, and precise correct answer.
- Video Tool Analysis: Table 7 compares accuracy across four Gemini models under No Search, Native Search, Native Search + Video Tool, and Agentic Multimodal Search.
E.1 Impact of Adding Video Processing Tool
Adding video processing to Native Search consistently improves accuracy, while Agentic Multimodal Search generally performs better through more proactive and targeted video retrieval.
- Adding a video processing tool improves Native Search accuracy by +1.0% to +8.5%, averaging +5.7% across agents.The gains indicate that video evidence is important for a substantial portion of MERRIN questions.
- Agentic Multimodal Search outperforms Native Search with video processing on three of four evaluated agents.The exception is Gemini 3 Flash, where Native Search with video processing reaches 36.8% versus 32.9% for Agentic Multimodal Search.
- Agentic Multimodal Search gains an advantage by proactively invoking video analysis and locating more relevant videos through a dedicated search tool.Native Search may fail to invoke video processing and sometimes retrieves irrelevant videos through built-in Google Search.