Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

1 Introduction

• Existing VLM benchmarks predominantly assess single-image understanding, overlooking comparison and reasoning across images.The paper identifies such abilities as important for applications including comparing shopping items, analyzing X-rays from different angles, and understanding temporal sequences.Source: §1 · p. 2
• MIRB evaluates perception, visual world knowledge, reasoning, and multi-hop reasoning across multiple images.Its tasks require comparing multiple input images to derive solutions.Source: §1 · p. 2
• Open-source VLMs trail GPT-4V substantially on multi-image reasoning, and GPT-4V itself struggles on MIRB.LLaVA performs comparably to GPT-4V on single-image reasoning and question answering, but the multi-image gap remains.Source: §1 · p. 2
• The paper contributes a benchmark, an evaluation of open- and closed-source models, and findings on multi-image reasoning challenges.The authors present MIRB as a testbed for evaluating diverse facets of multi-image understanding.Source: §1 · p. 2

2 Related Work

• LVLMs combine LLMs with a visual encoder and connection module, and visual instruction fine-tuning has supported multimodal perception and reasoning.The cited work describes progress in large vision-language models.Source: §2 · p. 2
• Many LVLM training datasets use single image-text pairs or text alone, while multi-image evaluation has predominantly remained qualitative.Some work trains models on interleaved image-text corpora such as MMC4 and OBELICS.Source: §2 · pp. 2–3
• Traditional and aggregated LVLM benchmarks generally test single-image inputs, covering tasks such as OCR, captioning, visual question answering, and broader model abilities.Examples include MME, MM-Vet, MMBench, SeedBench, and MMMU.Source: §2 · p. 3
• Multi-image benchmarks address narrower targets: Q-Bench low-level perception, Memontos temporal understanding, Seed-Bench-2 VQA, and DEMON demonstrative instruction following.The paper identifies concurrent work BLINK as most related and says MIRB covers a more diverse range of tasks and evaluation dimensions.Source: §2 · p. 3

3 Dataset and Task Categories

• MIRB covers multi-image reasoning, visual world knowledge, perception, and multi-hop reasoning, with questions designed to require comparing images.The benchmark focuses on reasoning across multiple image inputs.Source: §3 · p. 3, §3.1 · p. 5

3.1 Design of Subtasks

• MIRB's multi-image reasoning tasks include code understanding, matching plots to source code, visual analogy, and 3D scene understanding.The 3D scenes use multiple 2D views, where information such as the total object count may require comparing views with occlusions.Source: §3.1 · pp. 3–4
• The visual world knowledge dimension includes sightseeing questions about whether locations are in the same city or identifying a city, and food-ingredient comparisons against a text criterion.Sightseeing images are queried from Pixabay and manually filtered; ingredient-list images come from OpenFoodFact.Source: §3.1 · pp. 4–5
• Perception tasks test image-jigsaw reconstruction, counting objects across images, and matching attributes such as artistic style.The tasks use image patches, object annotations, and object-category and style annotations, respectively.Source: §3.1 · p. 5
• Multi-hop reasoning tasks include synthetic visual logic chains and looking up cited papers from screenshots of arXiv pages.Both require relating information across images to reach an answer.Source: §3.1 · p. 5
• MIRB uses independently sourced images rather than redundant frames from a video, and each question requires at least two images.Multi-hop questions can require up to 42 images; a question requires three images on average.Source: §3.1 · pp. 5–6

4 Experiments

• The evaluation comprehensively covers 12 state-of-the-art VLLMs across model families and sizes from 2.7B to 37B, including GPT4-V.Open-source models use greedy decoding; GPT4-V uses the “2024-05-13” API version.Source: §4 · p. 6
• On perception, the best open-source model performs close to GPT4-V, but open-source models lag GPT4-V by a great margin on reasoning.No open-source model reliably outperforms the random-chance baseline on visual world knowledge or multi-hop reasoning.Source: §4 · p. 6

5.1 Our benchmark is Not Solvable with Only One Image

• Models require multiple images to achieve reasonable performance on the tested 3D-scene, code-understanding, and synthetic visual-logic subsets.The analysis compared using all image inputs with using only one image; other subsets were excluded because they inherently require image comparison or use image-based options.Source: §5.1 · p. 7, Figure 3 · p. 8

5.2 Concatenated Images

• Most models perform worse when original images are concatenated into one larger image, while LLaVA improves under this encoding.The authors attribute the LLaVA exception to its single-image training and the reduced shift between training and testing input counts.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 LVLMs Understand Text Inputs Better than Images

• Text inputs are usually better than image inputs on text-understanding tasks, but the authors cannot disentangle perception difficulty from token-length effects.The comparison uses source code or OCR-extracted text directly as LLM input without a vision encoder.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4 Chain-of-Thought Does Not Help Reasoning Over Multiple Images

• Zero-shot chain-of-thought prompting yields only minor improvements or sometimes decreases performance on the tested reasoning subsets.The authors conclude that simple prompting techniques do not effectively address multi-image reasoning and that fundamental training improvements are needed.Source: §5.4 · p. 8

Conclusion and results

• Open-source models lag GPT-4V on multi-image scenarios, and no open-source model consistently exceeds random chance on visual world knowledge or multi-hop reasoning.The conclusion contrasts these results with open-source models performing well on single-image tasks.Source: §6 · p. 8

Analysis experiments

• Analysis experiments find that MIRB questions require reasoning across multiple images; concatenating images performs poorly, while text inputs aid reasoning more than image inputs.The conclusion also reports that chain-of-thought prompting brings no notable improvement and that models cannot reliably decode text in images.Source: §6 · pp. 8–9

Implications

• The authors argue MIRB highlights the need for model architectures and training datasets better suited to multi-image contexts and can serve as a research resource.They describe the benchmark as a roadmap for future research on more capable multimodal models.Source: §6 · p. 9

Limitations and broader impact

• MIRB cannot reflect every real-world use case; critical applications such as self-driving cars require additional attention to safety testing.Source: §6 · p. 9

Limitations, impact, and assets

• The checklist marks discussion of work limitations and potential negative societal impacts as included, and marks applicable asset attribution and licensing checks as completed.Source: Checklist · p. 13

Reproducibility

• The authors report that they will provide code, data, and instructions for reproducing the main experimental results in supplementary materials.Source: Checklist · p. 13

Experimental reporting

• Training details are marked not applicable because the authors did not train new models; experiments use fixed random seeds and are described as fully reproducible.Source: Checklist · p. 13

Human subjects

• Human-subjects and crowdsourcing items are marked not applicable in the checklist.Source: Checklist · p. 13

Access

• MIRB is available on HuggingFace, with evaluation code and documentation available on GitHub.Source: A Dataset Access and Copyright Statement · p. 14

Intended use

• MIRB is intended only to benchmark multi-image reasoning, and its score should not be the sole measure of a VLM's capability.Source: A Dataset Access and Copyright Statement · p. 14

Copyright

• The authors collected and filtered all benchmark images and accept responsibility for any copyright violations.Source: A Dataset Access and Copyright Statement · p. 14

Data sources

• Table 4 lists the data sources used to collect the benchmark images.Source: Table 4 · p. 14

• MIRB uses rendered code snippets and Blender-generated scene views alongside images from existing datasets and websites.Sources include Bitton et al. for visual analogy, COCO for counting, OpenFoodFacts for comparison, and ImageNet-R for attribute matching.Source: B Data Source · p. 14

• Current LVLMs show volatile MIRB performance, are sensitive to image presentation, and struggle to gather cross-image information such as symbols in synthetic logic-chain questions.Figure 4 provides qualitative examples illustrating model performance on MIRB.Source: C Qualitative Results · p. 14, Figure 4 · p. 15