MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

1 Introduction

• Existing multimodal LLM evaluations generally emphasize single-image understanding, while recent multi-image benchmarks remain limited in perspectives, relations, and robustness.The authors identify holistic integration and reasoning across multiple images as an overlooked evaluation need.Source: §1 · p. 2
• MUIRBENCH contains 11,264 images and 2,600 multiple-choice questions across 12 multi-image tasks and diverse relations, including temporal, ordered-page, and narrative relations.Questions and choices are derived from datasets or written manually by experts.Source: §1 · pp. 2–3
• Each question-answering instance is paired with an expert-annotated unanswerable counterpart with minimal differences, helping limit correct answers from vision or language shortcuts.Fine-grained labels, including image positions and image types, support detailed model analysis.Source: §1 · p. 3
• GPT-4o scores 68.0% accuracy and Gemini Pro 49.3%, respectively 25.1% and 43.8% below human performance; their unanswerable-question accuracy gaps are 26.8% and 21.5%.The evaluation covers 20 multimodal LLMs, including models that accept multi-image inputs and models originally designed for single-image inputs.Source: §1 · p. 3

2.1 Multimodal Understanding Benchmarks

• MUIRBENCH offers broader multi-image assessment than prior benchmarks, which largely target single images or limited relations and differ from multi-image understanding in video and in-context learning.Existing dedicated benchmarks MANTIS-Eval [25] and DEMON [31] cover limited relations or reasoning processes and lack robust evaluation.Source: §2.1 · p. 3

2.2 Multimodal Large Language Models

• Although early multimodal LLMs largely used single-image data and newer models train on interleaved images or show counting and comparison skills, diverse relational reasoning across independent multi-image inputs remains unexplored.MUIRBENCH is proposed to provide comprehensive evaluation and insights for improving models’ handling of realistic multi-image tasks.Source: §2.2 · pp. 3–4

3.1 Benchmark Overview

• 11,264 images and 2,600 multiple-choice questions form MUIRBENCH, spanning 12 tasks and 10 relation categories with an average of 4.3 images per instance.Each answerable instance is paired with an unanswerable counterpart that differs minimally; fine-grained labels record image positions and types.Source: §3.1 · p. 4
• The 12 tasks cover capabilities from action and attribute understanding to counting, ordering, scene understanding, visual grounding, and retrieval.Instances include 2–9 images in varied question positions and types, including slides, maps, medical images, animations, and 3D views.Source: §3.1 · pp. 4–5
• MUIRBENCH pairs every answerable question with a minimally changed unanswerable counterpart to test whether models recognize when queries cannot be answered.Unanswerable variants are created by replacing or reordering images, modifying questions, or changing options.Source: §3.1 · p. 5, Figure 5 · p. 5

3.2 Data Collection

• 40.8% of answerable data come from existing datasets, 21.7% from dataset derivations, and 37.5% from newly collected data.New data address underrepresented tasks, while multiple-choice questions support deterministic evaluation.Source: §3.2 · p. 5
• 40.5% of unanswerable variants modify options, 35.3% modify questions, and 24.2% replace or reorder images; each answerable instance receives one strategy.Source: §3.2 · pp. 5–6
• Automatic checks validate instance format, answers, metadata, image references, and image accessibility, while four experts filter ambiguous, unclear, or confusing instances.Source: §3.2 · p. 6

4.1 Experimental Setup

• Evaluation covers 20 recent multimodal LLMs designed for multi-image inputs and models originally designed for single-image inputs.The setup follows VLMEvalKit with temperature 0 and 10 retries; models without multiple-image input support receive concatenated images.Source: §4.1 · p. 6

4.2 Main Results

• Multi-image-input models can outperform single-image-input models even with fewer parameters, indicating that transfer from single-image training is non-trivial.The results suggest that multi-image training data and learning processes benefit multi-image understanding.Source: §4.2 · pp. 6–7
• Advanced multimodal LLMs average no better than 68% accuracy, while open-source models considering multiple images range from 23.73% to 44.50%.Model size shows no obvious correlation with performance, and some model-task results are at or below random guessing.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Analysis

• Among 100 sampled GPT-4o errors, image-detail failures account for 26%, object-counting or reasoning for 20%, and logical reasoning for 18%.Other error categories are identifying the same object across scenes (14%) and inferring intent from image sequences (12%).Source: §4.3 · p. 8

4.2 Main Results

• Models perform relatively better on image-text matching, visual retrieval, and diagram understanding than on multi-image ordering and visual grounding.Ordering and grounding require interpreting the full multi-image context and conducting more complex reasoning across images and modalities.Source: §4.2 · p. 6, Figure 7 · p. 9

4.3 Analysis

• All studied models show a severe performance drop on unanswerable instances compared with answerable ones.Models often avoid abstaining on unanswerable questions, and the pairwise design helps make MUIRBENCH assessment more reliable.Source: §4.3 · pp. 7–8, Figure 8 · p. 9
• On unanswerable questions, models may hallucinate a likely-looking option instead of abstaining.This behavior appears in examples from cartoon understanding, diagram understanding, visual grounding, and visual retrieval.Source: §4.3 · p. 9

• Experiments with 20 multimodal LLMs revealed substantial limitations in handling multi-image scenarios and performance deficits compared with human accuracy.Source: §5 · p. 9
• The evaluated models struggled more with unanswerable questions in MUIRBENCH.Source: §5 · p. 9
• MUIRBENCH offers a rigorous framework for assessing multi-image understanding and encourages models to synthesize and reason across multiple visual sources.Source: §5 · p. 9

A.1 Dataset Statistics

• MUIRBENCH includes image types from common photography to medical, slide, drone, and satellite imagery.Figure 10 presents the distribution by image type.Source: §A.1 · p. 15
• Instances range from two to nine images, and Figure 11 presents their distribution by image count.Source: §A.1 · p. 15
• Image positions include the beginning, middle, or end of a question, options, and mixtures of these positions.Figure 12 presents the distribution by image position.Source: §A.1 · p. 15

A.2 Dataset Curation Details

• MUIRBENCH combines sampled existing datasets (40.8%), derived datasets (21.7%), and four new datasets targeting underrepresented tasks, relations, and image types (37.5%).Existing and derived sources are sampled at up to 200 test examples per dataset; derivations reformat binary or open QA and convert single-image MCQA into multi-image MCQA. The new datasets are HistoricalMap, UnivBuilding, PubMedMQA, and SciSlides.Source: §A.2 · pp. 15–16
• Each answerable instance receives one minimally changed unanswerable counterpart, created by altering images (24.2%), questions (35.3%), or options (40.5%), doubling the data into balanced answerability classes.Image alterations replace or reorder images; question changes make the question incompatible with images and options, while option changes create no correct answer.Source: §A.2 · p. 17
• Metadata covers image relations, tasks, image types, image counts, and positions, with relations manually annotated, missing task and type labels filled manually, and counts and positions detected automatically.Image relations are emphasized as crucial to multi-image reasoning and rarely annotated in existing data.Source: §A.2 · p. 17
• Quality control combines rule-based automatic checks with manual instance review, retaining 86.3% after filtering ambiguous queries, unclear images, and other errors.Automatic checks verify format, answers, metadata, image-placeholder coreference, redundancy, and image accessibility.Source: §A.2 · p. 17

A.3 Multi-image Relations

• The taxonomy defines 10 relations across temporal, sequential, complementary, visual-similarity, multiview, cropped, narrative, and independent-image cases.Its categories are Temporal Relation, Ordered Pages, Complementary Relation, Cropped/Zoomed Images, Narrative, Scene-Multiview, Object-Multiview, Overall Similarity, Partial Similarity, and Independent Images.Source: §A.3 · p. 17

A.4 Human Evaluation Protocol

• Two domain experts conduct human evaluation, with each answerable instance and its unanswerable counterpart randomly assigned to different experts.The evaluation interface is shown in Figure 13.Source: §A.4 · p. 17, Figure 13 · p. 18

B Baseline Models

• The evaluation uses the latest and best-performing available checkpoint for most model families.The authors note GPT-4 performance may change as its specific checkpoint is updated.Source: B Baseline Models · p. 18
• The baselines include GPT-4, Gemini Pro [58], and multi-image-finetuned Mantis variants [25].GPT-4 is tested with gpt-4-turbo and gpt-4o; Gemini uses Gemini 1.0 Pro Vision; Mantis includes Idefics2, clip-llama3, and siglip-llama3 (all 8B).Source: B Baseline Models · p. 18
• Other models that accept multiple images include VILA, Idefics [28], Emu2 [55], and OpenFlamingo [2].The list also includes LLaVA variants, which were designed for single-image input; the authors concatenate the images in order.Source: B Baseline Models · pp. 18–19
• The remaining baselines include Yi-VL-6B7, MiniGPT-4-v2 [8], and CogVLM.MiniGPT-4-v2 uses EVA [14], LLaMA2-chat (7B) [61], and a linear projection layer; CogVLM adds a trainable visual expert module to attention and FFN layers, using EVA-CLIP [56] and Vicuna [69].Source: B Baseline Models · p. 19

C.1 Model Prompts

• Following [41], the prompt combines a question, choices, an answer-format hint, and an answer prefix, with images inserted into the text.The prompt is designed to form a coherent text-and-image input.Source: §C.1 · p. 19
• The template lists four labeled options and requests the correct option letter directly, then begins the answer with “Answer:”.The hint gives A, B, C, or D as examples of the requested format.Source: §C.1 · p. 19

C.2 Evaluation Tool

• Following [66], the rule-based tool first detects a valid option index in the model output as the exact answer.This is the first step in the automatic extraction process.Source: §C.2 · p. 19
• If no direct answer appears, the tool matches the model output against the content of each option.Source: §C.2 · p. 19
• If neither an index nor an option-content match is found, the tool randomly selects an option as the answer.Source: §C.2 · p. 19
• When the output contains multiple valid answers, the tool chooses the first one that appears as the final answer.Source: §C.2 · p. 19

D Error Analysis

• Images in answer options yield the highest accuracy, while images in the middle of questions yield the highest error rates across the three analyzed models.The authors suggest that interruption of context processing or less training exposure to middle-position images may contribute.Source: Figure 14 · p. 19, D Error Analysis · p. 20
• All three models perform relatively better when only the question is changed to make it incompatible with the original images and options.Source: Figure 15 · p. 19, D Error Analysis · p. 20
• When the correct option is removed, all three models become confused and fail to choose “none of the other options.”Source: Figure 15 · p. 19, D Error Analysis · p. 20
• For unanswerable instances made by reordering or replacing images, GPT-4o performs much better than the other analyzed models.Source: Figure 15 · p. 19, D Error Analysis · p. 20

E.1 Limitation and future work

• MUIRBENCH evaluates 2D images and multimodal LLMs; future work could extend it to 3D, more tasks and relation categories, other foundation models, and additional unanswerable-instance strategies.The authors also hope the benchmark guides robust and faithful multimodal evaluation.Source: §E.1 · p. 20

E.2 Societal impacts

• The English-only tasks may bias multilingual research, and multimodal models could be misused to generate harmful artifacts; curated data contain no personal information or offensive content.The authors encourage further research on multimodal safety.Source: §E.2 · p. 20

F License

• MUIRBENCH data are released under CC-BY 4.0, while individual source datasets retain their original licenses.The paper enumerates those source-specific licenses.Source: F License · p. 20

G.1 Dataset Documentation and Format

• Each MUIRBENCH entry includes its index, task, question, options, answer, image relation and type, images, and counterpart-instance index.Full documentation is available on the project page.Source: §G.1 · p. 21

G.2 Links and Maintenance Plan

• MUIRBENCH is hosted on Hugging Face, where its license and metadata are available; the benchmark will be maintained and updated, with evaluation code and outputs provided for reproduction and analysis.Source: §G.2 · p. 21

G.3 Author Statement

• The authors accept responsibility for rights violations arising during data collection and commit to taking appropriate action when needed.Source: §G.3 · p. 21

G.4 Intended Uses

• The dataset is intended for academic purposes only and is not for commercial use.Source: §G.4 · p. 21