Source-linked AI summary
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, Muhao Chen
TL;DR
Multimodal LLMs have been evaluated mainly on single images, leaving robust multi-image understanding insufficiently assessed. MUIRBENCH addresses this gap with a broad pairwise benchmark of answerable and minimally altered unanswerable instances, and finds substantial limitations even in leading models. The benchmark supports more holistic evaluation of models’ ability to synthesize and reason across images.
Problem
Existing evaluations largely focus on single-image understanding, leaving comprehensive evidence about multimodal LLMs’ multi-image reasoning limited.
Method
MUIRBENCH evaluates 12 tasks across 10 multi-image relation categories using 11,264 images, 2,600 questions, and paired unanswerable variants with minimal differences.
Results
68.0% and 49.3% accuracy were achieved by GPT-4o and Gemini Pro, respectively, while multimodal LLMs performed substantially worse on unanswerable questions and single-image-trained models generalized poorly.
Takeaways & Limitations
MUIRBENCH provides a rigorous testbed for assessing holistic multi-image understanding and encouraging models to transcend single-image limitations.
Takeaways & Limitations
The benchmark focuses on 2D images, does not cover all strategies for creating unanswerable instances, and evaluates multimodal large language models rather than a broader set of vision-language foundation models.
Abstract
from arXiv · showhide
We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.
1 Introduction
MUIRBENCH addresses the gap between strong single-image multimodal performance and the harder problem of integrating information across multiple images. Its pairwise design and evaluation results expose substantial limitations in current multimodal LLMs, including on unanswerable questions.
- Single-image benchmarks overlook the richer tasks of integrating and reasoning across multiple images.
- MUIRBENCH contains 11,264 images and 2,600 multiple-choice questions spanning 12 multi-image understanding tasks and diverse image relations.
- Each question-answering instance is paired with an expert-annotated unanswerable counterpart featuring minimal differences, reducing vision and language shortcut risks.
- 68.0% and 49.3% accuracy were achieved by GPT-4o and Gemini Pro, respectively, while remaining 25.1% and 43.8% below human performance.
- GPT-4o and Gemini Pro showed 26.8% and 21.5% accuracy gaps between answerable and unanswerable questions.
2 Related Work
Existing multimodal benchmarks largely emphasize single-image understanding, while multi-image understanding requires integrating spatial and contextual cues across varied visual sources. MUIRBENCH is proposed as a comprehensive evaluation and improvement-oriented testbed for this capability.
- Most existing multimodal benchmarks focus on single-image scenarios and provide only limited coverage of multi-image capabilities.
- Multi-image understanding differs from video understanding and in-context learning by requiring integration of spatial and contextual cues across varied perspectives, settings, and moments.
- Early multimodal LLMs were mainly trained on single-image datasets, while newer work explores interleaved image-text pretraining and multi-image instruction tuning.
- MUIRBENCH is proposed to provide comprehensive evaluation and insights for improving multimodal LLMs on realistic multi-image tasks.
3 MUIRBENCH
MUIRBENCH combines broad task and relation coverage with a robust pairwise design for evaluating holistic multi-image understanding. It uses answerable questions, minimally altered unanswerable counterparts, expert-defined labels, and quality-control procedures.
- Benchmark Overview: MUIRBENCH contains 11,264 images and 2,600 multiple-choice questions, averaging 4.3 images per instance.
- Benchmark Overview: The benchmark covers 12 multi-image tasks and 10 relation categories, with fine-grained labels for image positions and image types.
- Comprehensive Multi-Image Evaluation: Each task represents 2.5% to 17.8% of the benchmark, providing coverage across diverse multi-image understanding abilities.
- Comprehensive Multi-Image Evaluation: Tasks include chronological action understanding, attribute similarity, cartoon understanding, counting, and diagram understanding across multiple images.
- Comprehensive Multi-Image Evaluation: Images represent narrative, ordered-page, temporal, and multiview relations, with two to nine images placed at different positions in each instance.
- Robust Evaluation: Answerable instances are paired with minimally different unanswerable counterparts to test whether models recognize when queries cannot be answered.
- Data Collection and Quality Control: The dataset combines existing data, derived data, and newly collected data, while automatic checks and expert review support annotation quality.
- Robust Evaluation: Unanswerable variants are created by disrupting image relations, making questions incompatible with the inputs, or replacing options so no correct answer exists.
4 Experiments
MUIRBENCH evaluates 20 multimodal LLMs across diverse multi-image tasks and shows that current models remain far from reliable multi-image understanding. Performance varies by task, input format, answerability, and model training, with unanswerable questions and cross-image reasoning posing particular challenges.
- Evaluation setup: 20 recent multimodal LLMs were evaluated on MUIRBENCH, including models designed for multi-image and single-image inputs.The evaluation used standard temperature-0 inference settings and concatenated images for models without native multi-image support.
- Overall results: 23.73%–44.50% average accuracy was achieved by open-source multimodal LLMs, with no obvious correlation between model size and performance.Some model-task results were at or below random guessing, suggesting that training data and training processes matter for multi-image capability.
- Performance by task: Image-text matching, visual retrieval, and diagram understanding were relative strengths, whereas multi-image ordering and visual grounding were more challenging.The harder tasks require understanding the whole multi-image context and performing more complicated reasoning across images and modalities.
- Input capability: Multi-image-input models generally outperformed single-image-input models, showing that generalizing from single-image training to multi-image inference is non-trivial.The comparison indicates benefits from multi-image training data and learning processes, even for models with fewer parameters.
- Answerability: All studied models suffered a severe performance drop on unanswerable counterparts and often avoided abstention by selecting an apparently plausible option.This behavior motivates evaluating answerable and minimally changed unanswerable instances together for more reliable assessment.
- Error analysis: 26% of GPT-4o’s sampled errors involved failure to capture image details, followed by object counting or reasoning at 20% and logical reasoning at 18%.Other errors involved identifying the same object across scenes at 14% and inferring intents from image sequences at 12%.
5 Conclusion
MUIRBENCH provides a robust evaluation of multimodal LLMs’ multi-image understanding. Results across 20 models show substantial limitations, especially relative to humans and on unanswerable questions, motivating more holistic visual reasoning across multiple sources.
- Conclusion: MUIRBENCH evaluates multi-image understanding and reveals substantial limitations in 20 multimodal LLMs, including significant deficits relative to human accuracy.The benchmark is intended to provide a rigorous framework for assessing synthesis and reasoning across multiple visual sources.
- Conclusion: Models struggled more with unanswerable questions, reinforcing the importance of evaluation settings that test whether systems can reason across multiple visual sources.The conclusion frames this benchmark as encouragement for developing more holistic visual comprehension.
Appendices
The appendix documents MUIRBENCH’s broad coverage of image types, image counts, and image positions, supporting analyses of how these factors relate to model behavior.
- Dataset statistics: MUIRBENCH covers image types from ordinary photography to medical images, slides, and drone or satellite imagery.The benchmark also varies the number of images per instance from two to nine.
- Dataset statistics: Instances contain between two and nine images, while image-position distributions are separately analyzed.The reported statistics distinguish the number and positional arrangement of images within benchmark instances.
A.2 Dataset Curation Details
MUIRBENCH combines diverse answerable data, expert-created unanswerable counterparts, fine-grained metadata, and quality-control procedures to support diagnostic multi-image evaluation.
- Answerable data collection: Answerable data combine existing datasets, derived data, and newly collected questions covering multiple tasks and image relations.The multiple-choice format is used to provide deterministic results and fine-grained diagnostic evaluation.
- Answerable data collection: 1300 answerable instances comprise 531 existing examples, 282 derived examples, and 487 new examples.These correspond to 40.8%, 21.7%, and 37.5% of the answerable data, respectively.
- New data: New datasets fill gaps in geographic understanding, multiview relations, and specialized image types such as medical images and presentation slides.Examples include HistoricalMap, UnivBuilding, PubMedMQA, and SciSlides.
- Unanswerable data: Unanswerable counterparts are created by minimally replacing or reordering images, modifying questions, or replacing options so that no correct answer remains.Each answerable instance receives one of these transformations.
- Metadata annotation: Fine-grained metadata annotate image relations, tasks, image types, image counts, and image positions for diagnostic weakness analysis.Image relations receive manual annotation because they are important for multi-image reasoning yet rarely annotated in existing data.
- Quality control: Quality control combines automatic checks with manual examination to remove invalid, ambiguous, unclear, or inaccessible instances.Automatic checks verify formats, answers, metadata, image references, and accessibility.
- Multi-image relations: MUIRBENCH includes 10 multi-image relation categories, including temporal, ordered-page, complementary, cropped or zoomed, narrative, scene-multiview, and object-multiview relations.These relations encode how multiple images jointly convey time, sequence, context, perspective, or shared content.
- Human evaluation: Two domain experts independently evaluate answerable instances and their unanswerable counterparts through a dedicated human-evaluation interface.Random assignment of instances to different experts is used to support fair evaluation.
B Baseline Models
The benchmark evaluates a broad set of multimodal LLM baselines, including models designed for multi-image input and models originally designed for single-image input. The evaluated families span proprietary and open-source systems with varied architectures and sizes.
- Baseline coverage: 20 multimodal LLMs are evaluated, including models designed for multi-image inputs and models originally designed for single-image inputs.For most model families, the authors use the latest and best-performing available checkpoint.
- Single-image models: The baselines also include LLaVA variants designed for single-image input, with all images concatenated in order for evaluation.The included LLaVA versions span v1.5, NeXT, internLM, and xtuner models with sizes from 7B to 34B.
C Experiment Setting Details
The experiment uses a standardized multiple-choice prompt that combines the question, answer options, response-format hint, and answer prefix, with images inserted into the text. A rule-based tool extracts or assigns the final option.
- Prompt format: Each prompt contains the question, four answer choices, an answer-format hint, and an answer prefix.The hint asks models to directly provide the correct option letter: A, B, C, or D.
- Prompt format: Images are inserted into the text to form a coherent multimodal prompt.The resulting prompt interleaves image inputs with the textual question and choices.
- Answer extraction: The evaluation tool first detects a valid option index, then matches the output to option content, and randomly selects an option if no match is found.When multiple valid answers appear, the first detected answer is used.
D Error Analysis
Error analyses examine how image position and unanswerable-instance type relate to model accuracy. Models perform best when images are options, struggle when images appear mid-question, and are especially confused when the correct option is removed.
- Image position: Image position correlates with error rates across GPT-4o, GeminiProVision, and Mantis-8B-Idefics2.Accuracy is highest when images are positioned in options, whereas error rates are highest when images are in the middle of questions.
- Unanswerable types: All three analyzed models perform relatively better when only the question is changed to become incompatible with the original images and options.This compares one unanswerable construction with other types analyzed in the figure.
- Unanswerable types: All three models are confused when the correct option is removed and fail to choose “none of the other options.”The result concerns unanswerable instances created by removing the correct choice.
- Unanswerable types: GPT-4o performs much better than the other models on unanswerable instances created by reordering or replacing images.Performance on these instance types is described as divergent across the three models.
E.1 Limitation and future work
The paper limits its scope to 2D, English-language multimodal-LLM evaluation and does not cover every strategy for constructing unanswerable instances. It releases the benchmark data and documentation under stated licensing and accountability conditions.
- Limitations and future work: The benchmark focuses on 2D images, leaving extension to 3D problems, additional multi-image tasks, and more relation categories for future work.The authors also state that their unanswerable-instance strategies do not cover all possible construction strategies.
- Limitations and future work: The evaluation focuses on multimodal large language models, while future work could include additional vision-language foundation models such as Unified-IO 2 and Chameleon.This defines the current model-family boundary rather than a claim about those future models.
- Societal impacts: All 12 tasks are in English, which could induce bias in multilingual research settings.The authors frame this as a limitation of the benchmark’s language coverage.
- Release and reproducibility: MUIRBENCH metadata includes task, question, options, answer, image relation, image type, images, and counterpart-instance identifiers.The full documentation is hosted on the project page, and the benchmark is maintained on Huggingface/Datasets with evaluation code and outputs intended to support reproduction.
- Data governance: The authors accept responsibility for rights violations during data collection and state that curated data contain no personally identifiable information or offensive content.They also state that the dataset is for academic purposes only and not for commercial use.