PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Chaoyou Fu, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, Hongsheng Li

arXiv:2409.12959v2cs.CVcs.AIcs.CLcs.IR

TL;DR

Current AI search engines are limited to text, while a framework for LMMs to function as multimodal search engines remains largely unexplored. MMSearch-Engine and MMSearch provide a pipeline and benchmark for evaluating LMM search; GPT-4o achieves the best overall performance, and MMSearch-Engine with GPT-4o and Claude 3.5 Sonnet surpasses Perplexity Pro on the end-to-end task.

  • Problem

    Current AI search engines process textual queries and web content, while a framework for LMMs to function as multimodal search engines remains largely unexplored.

    Source: §1 · pp. 2–3

  • Method

    MMSearch-Engine enables zero-shot multimodal search by LMMs, and MMSearch benchmarks their search capabilities with 300 queries across 14 subfields.

    Source: §1 · p. 3, §4 · p. 13

  • Results

    GPT-4o achieves the best overall performance across tasks, and MMSearch-Engine with GPT-4o and Claude 3.5 Sonnet surpasses Perplexity Pro on the end-to-end task.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Step-by-step evaluation finds current models fall short of human-level search proficiency and identifies failure patterns in key search steps for future improvement.

    Source: §4 · p. 13

    Error analysis identifies requery specificity and efficiency failures, plus summarization errors in text reasoning, image-text aggregation, image reasoning, hallucination, and output format.

    Source: §3.3.2 · p. 12

Abstract

from arXiv · show

The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: https://mmsearch.github.io

1 INTRODUCTION

The paper introduces MMSEARCH-ENGINE and the MMSEARCH benchmark to evaluate LMMs as multimodal search engines. GPT-4o achieves the best overall performance, and MMSEARCH-ENGINE with GPT-4o or Claude 3.5 Sonnet surpasses Perplexity Pro end-to-end.

Motivation

  • Text-only search engines cannot process image queries or interpret interleaved text-image web content, while a multimodal search framework for LMMs remains largely unexplored.These limitations constrain query scenarios and information seeking, motivating evaluation of LMMs' potential for multimodal search.Source: §1 · pp. 2–3

MMSEARCH-ENGINE

  • MMSEARCH-ENGINE enables multimodal search by using Google Lens for image queries, presenting results in text and visual formats, and running requery, reranking, and summarization.Its step-wise evaluation covers those three core tasks alongside an end-to-end task, with the final score weighted by end-to-end and core-task results.Source: §1 · p. 3

MMSEARCH benchmark

  • MMSEARCH contains 300 queries across 14 subfields and two areas—latest news and rare knowledge—plus 2,901 unique images.News queries reflect information current at August 2024; knowledge queries were selected because leading models such as GPT-4o and Claude 3.5 could not answer them.Source: §1 · p. 3

Experiments and analysis

  • GPT-4o achieves the best overall performance, and MMSEARCH-ENGINE with GPT-4o and Claude 3.5 Sonnet surpasses Perplexity Pro on the end-to-end task.Error analysis finds that weak requery and reranking limit website selection and answer extraction; initial ablation results favor scaling test-time computation over model size.Source: §1 · p. 3
  • Error analysis finds that LMMs struggle to understand requery and extract useful information from text and images during summarization.The authors identify five error types for each of requery and summarization; initial results suggest test-time computation scaling outperforms model-size scaling on this task.Source: §1 · p. 3

2 MMSEARCH

MMSEARCH pairs a multimodal search pipeline with a benchmark, step-wise and end-to-end evaluation, and a news component designed to evolve as model knowledge changes.

2.1 MMSEARCH-ENGINE: A MULTIMODAL AI SEARCH ENGINE PIPELINE

  • The pipeline uses Google Lens results and website screenshots to support three stages: requery, rerank, and answer summarization.It trims blank screenshot areas and retrieves up to 2K text tokens relevant to the requery before providing website content to the LMM.Source: §2.1 · pp. 4–5

2.2 DATA COMPOSITION AND COLLECTION

  • MMSEARCH contains 300 queries across News and rare Knowledge topics in 14 subfields, with questions collected and reviewed to ensure they are answerable from retrieved websites.News queries concern events from the collection period, while Knowledge questions are verified to exceed the capabilities of state-of-the-art LLMs such as GPT-4o or Claude 3.5 Sonnet.Source: §2.2 · pp. 5–6

2.3 EVALUATION PROTOCOL

  • The evaluation combines end-to-end F1, requery similarity, rerank relevance, and summarization F1, weighting end-to-end performance at 75% and the intermediate tasks at 25%.The intermediate weights are 5% for requery, 10% for rerank, and 10% for summarization; step-wise scores help distinguish errors across pipeline stages.Source: §2.3 · p. 6

2.4 BENCHMARK EVOLUTION

  • News questions span May 1 to August 31, 2024, beyond the stated or examined knowledge limits of evaluated models, and the benchmark will be updated if future training data overlaps.The Knowledge area remains static, while the News area is dynamic; the authors use this temporal gap to make evaluation reflect search capability rather than pre-existing knowledge.Source: §2.2 · p. 5, §2.4 · p. 7, Figure 5 · p. 7

3 EXPERIMENT

The evaluation finds that MMSEARCH-ENGINE with GPT-4o leads overall, while multimodal search remains constrained by weak requery and reranking. Error analyses identify failures in both search steps and multimodal content understanding, and test-time computation substantially improves end-to-end performance.

3.1 EXPERIMENT SETUP

  • The setup tests Perplexity Pro, closed-source LMMs, and open-source LMMs; open-source models receive no search-task fine-tuning, and requery uses three examples while other prompts are zero-shot.The retrieval setting uses K=8; full-page screenshots are split into 512-pixel segments, with a maximum of ten segments.Source: §3.1 · pp. 7, 9

3.2 EXPERIMENTAL ANALYSIS

  • GPT-4o leads overall at 62.3%; Qwen2-VL-72B leads open-source models but trails by 9.6%, with the gap reaching 11.3% end-to-end and 20.1% for 7B models.Closed-source LMMs consistently outperform open-source models on the final score.Source: Table 2 · p. 9, §3.2 · p. 10
  • For mPlug-Owl3, AnyRes improves overall score by 1.8%, end-to-end by 2.7%, and rerank by 0.2%, but trails LowRes on requery by 0.8% and summarization by 1.7%.The authors conclude that image OCR and perception quality do not bottleneck search performance; weak inherent search capabilities appear more limiting.Source: §3.2 · pp. 9–10
  • The summarization/end-to-end gap ranges from GPT-4o’s 2.7% to LLaVA-OneVision-7B’s 23.9%, reflecting the effect of earlier requery and rerank quality on end-to-end inputs.Most open-source models have gaps above 14%, while closed-source models are below 10%; Mantis is an exception at 3.4%, and Qwen2-VL-72B’s gap is 10.5%.Source: §3.2 · p. 10
  • MMSEARCH-ENGINE outperforms Perplexity Pro end-to-end with the same model and even with Qwen2-VL-72B, supporting the value of its image-search step.The authors attribute Perplexity’s weaker performance to rudimentary image search that can miss key objects and relevant information.Source: §3.2 · pp. 10–11

3.3.1 ERROR ANALYSIS OF END-TO-END TASK

  • GPT-4o’s main end-to-end errors are reranking and summarization, whereas Qwen2-VL-7B shows similar frequencies across all four error types.The categories are incorrect requery, selecting a site without the answer, failing to extract an answer present in site content, and format deviations; the authors associate Qwen2-VL’s informal errors with weaker instruction following.Source: Figure 7 · p. 11, §3.3.1 · p. 11

3.3.2 ERROR ANALYSIS OF REUQERY AND SUMMARIZATION TASK

  • Requery errors include underspecified, inefficient, unchanged, irrelevant, or image-result-ignoring queries; summarization errors include reasoning, image-text aggregation, hallucination, and format failures.The analysis links these patterns to difficulty understanding the task, aggregating available information, and extracting multimodal content; hallucination is labeled (Huang et al., 2023).Source: §3.3.2 · p. 12

3.4 SCALING TEST-TIME COMPUTE VS SCALING MODEL SIZE

  • With 25 test-time outputs, LLaVA-OneVision-7B scores 55.2% end-to-end, above its original 29.6%, LLaVA-OneVision-72B’s 44.9%, and GPT-4V’s 52.1%.The best-of-N-like procedure samples five requeries, five website selections, and five answers, then uses the highest answer F1 score.Source: §3.4 · p. 13

4 CONCLUSION

The paper introduces MMSEARCH-ENGINE, a zero-shot multimodal search pipeline, and MMSEARCH, a benchmark for evaluating LMM search capabilities. Evaluations find that current models remain below human-level search proficiency, with error analysis identifying failure patterns across key search steps.

4 CONCLUSION

  • Current LMMs still fall short of human-level search proficiency, and error analysis identifies failure patterns in key search-process steps.The authors present these patterns as insights for future improvements in LMM search ability.Source: §4 · p. 13
  • MMSEARCH-ENGINE is a streamlined pipeline that enables zero-shot LMMs to perform multimodal searches.Source: §4 · p. 13
  • MMSEARCH comprises 300 queries across 14 subfields to comprehensively assess multimodal search capabilities.Source: §4 · p. 13
  • The evaluation analyzes LMM search abilities step by step, enabling deeper understanding of their limitations.Source: §4 · p. 13

APPENDIX OVERVIEW

The appendix covers related work, additional experimental details, further dataset details, and qualitative examples.

Section A

Section B

Sections C–D

A RELATED WORK

Prior work advances LMMs across modalities and RAG-based retrieval, but multimodal search and its evaluation remain largely unexplored; MMSearch addresses this gap with a benchmark and search pipeline.

Large Multimodal Models

  • LMMs span image, 3D, and video; GPT-4o and Claude 3.5 show strong visual ability, yet multimodal search remains unexplored.Open-source systems and later work expand visual instruction tuning, high-resolution inputs, multiple vision encoders, and multi-image perception.Source: A RELATED WORK · p. 19

Large models with Retrieval Augmented Generation (RAG)

B ADDITIONAL EXPERIMENTAL DETAILS

This section details the evaluated model selection, a screenshot-slimming procedure, and the prompts used for multimodal search tasks.

Model Sources

Full-page Screenshot Slimming

  • Sobel gradients identify blank screenshot areas, which are iteratively removed to reduce image size while retaining critical document content.The figure caption describes the resulting size reduction as occurring without information loss.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 20, Figure 9 · p. 20

Input Prompts of LMM for Response Generation

C MORE DATA DETAILS · C.1 SUBFIELD DEFINITION

The data details divide news into eight subfields spanning specialized and general topics, and define a knowledge area focused on rare information that many LMMs fail to answer. The listed domains include fact-checking false premises and topics such as architecture, arts, fashion, astronomy, anime, and automobiles.

C.1 SUBFIELD DEFINITION

  • News covers eight subfields, ranging from everyday events and entertainment to scientific discoveries and financial analysis, to assess models across diverse domains.The subfields are Traditional Sports, e-Sports, Technology, Paper, Entertainment, Finance, General News, and False Premise.Source: §C.1 · p. 24
  • Traditional Sports and e-Sports cover athletic competitions and competitive gaming, including scores, standings, player or team information, rankings, and tournaments.The definitions also include transfers, streaming viewership statistics, and sporting analysis across leagues and countries.Source: §C.1 · p. 24
  • Technology and Paper cover technology developments and AI research publications, including product launches, cybersecurity, method explanations, figure understanding, and experiment settings.Technology also encompasses software updates and artificial intelligence advancements.Source: §C.1 · p. 24
  • Entertainment, Finance, and General News span popular culture, markets and economic policy, and broad local or global topics such as climate and lifestyle.Entertainment includes video games; finance includes stock prices, earnings, financial statements, and regulatory news.Source: §C.1 · p. 24
  • False Premise tests fact-checking of misinformation or incorrect query assumptions, and every answer in this subfield is “invalid question.”General News covers topics that do not fit neatly into a specialized news subfield.Source: §C.1 · p. 24
  • The knowledge area targets rare general knowledge that most LMMs fail to answer, and its listed subfields include architecture, arts, fashion, astronomy, anime, and auto.The passage introduces five subfields but lists six; their scope includes buildings and styles, creative works, clothing, celestial objects, Japanese animation, and automobiles.Source: §C.1 · p. 24

D QUALITATIVE EXAMPLES

The qualitative examples compare models’ responses and intermediate results in the end-to-end task.

D QUALITATIVE EXAMPLES

Related papers

Loading 2409.12959v2…