PaperlayerView exact revision on arXiv

Source-linked AI summary

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu

arXiv:2407.15272v2cs.CV

TL;DR

Most MLLMs and benchmarks focus on single images, leaving their abilities with realistic multiple-image inputs underexplored. MIBench evaluates those abilities across three scenarios, 13 tasks, and 13K annotated samples. Its evaluation finds significant multi-image shortcomings, including in fine-grained perception, reasoning, knowledge-seeking, and in-context learning.

  • Problem

    Most existing MLLMs and benchmarks focus on single-image inputs, while realistic multimedia commonly interleaves multiple images and text and existing multi-image benchmarks have limited evaluation dimensions and samples.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    MIBench evaluates MLLMs across MII, MKS, and MIC using 13 tasks and 13K annotated samples.

    Source: §1 · p. 2

  • Results

    Evaluation reveals that current MLLMs, especially open-source models, face significant challenges in fine-grained perception and multi-image reasoning, perform poorly in MKS, and have room to improve in MIC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The annotated MIBench data is publicly available to support further research on improving MLLMs’ multi-image abilities.

    Source: §5 · p. 9

    Because of current MLLM input-length limits, MII and MKS samples contain 2 to 8 images, although real-world scenarios may involve more.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Built on the power of LLMs, numerous multimodal large language models (MLLMs) have recently achieved remarkable performance on various vision-language tasks. However, most existing MLLMs and benchmarks primarily focus on single-image input scenarios, leaving the performance of MLLMs when handling realistic multiple images underexplored. Although a few benchmarks consider multiple images, their evaluation dimensions and samples are very limited. In this paper, we propose a new benchmark MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in multi-image scenarios. Specifically, MIBench categorizes the multi-image abilities into three scenarios: multi-image instruction (MII), multimodal knowledge-seeking (MKS) and multimodal in-context learning (MIC), and constructs 13 tasks with a total of 13K annotated samples. During data construction, for MII and MKS, we extract correct options from manual annotations and create challenging distractors to obtain multiple-choice questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and transform the original datasets into in-context learning formats. We evaluate several open-source and closed-source MLLMs on the proposed MIBench. The results reveal that although current models excel in single-image tasks, they exhibit significant shortcomings when faced with multi-image inputs, such as limited fine-grained perception, multi-image reasoning and in-context learning abilities. The annotated data of MIBench is available at https://huggingface.co/datasets/StarBottle/MIBench.

1 Introduction

Most MLLMs and benchmarks focus on single-image inputs, while realistic multimedia often interleaves multiple images and text. MIBench evaluates multi-image abilities across three scenarios and 13 tasks, and its results reveal substantial shortcomings in current models.

1 Introduction

  • MIBench is a large-scale benchmark with three multi-image scenarios, 13 tasks, and 13K samples.The scenarios are Multi-Image Instruction (MII), Multimodal Knowledge-Seeking (MKS), and Multimodal In-Context Learning (MIC).Source: §1 · p. 2
  • MII tests perception, comparison, and reasoning across images; MKS supplies interleaved image-text knowledge for questions that may concern one image or no image; MIC conditions responses on multimodal demonstrations.Source: §1 · p. 2
  • MII and MKS comprise 9K multiple-choice questions with task-specific distractors, while MIC contains 4K short-answer questions built from four datasets and demo samples.MKS also includes sampled images and associated texts as external knowledge.Source: §1 · p. 2
  • Current MLLMs, especially open-source models, struggle with fine-grained perception and multi-image reasoning; they also perform poorly in MKS and have room to improve in MIC.The evaluation includes both closed-source and open-source models.Source: §1 · p. 2

2 Related Work

Although multimodal models and benchmarks have developed, comprehensive evaluation of multi-image abilities remains lacking. MIBench addresses this gap with coverage of three scenarios and 13 tasks.

2.1 Multimodal Large Language Models

  • Prior work has extended LLMs to process visual and textual information, but a comprehensive benchmark for MLLM multi-image abilities is lacking.The passage states that this gap limits exploration of model potential and hinders field development.Source: §2.1 · pp. 2–3

2.2 MLLM Benchmarks

  • Existing benchmarks primarily assess single-image inputs, while recent multi-image benchmarks cover limited tasks or scale and may rely on GPT-4 scoring or manual annotation.Sparkles-Eval focuses on dialogue and uses GPT-4 scoring; Mantis-Eval and SEED-Bench2 cover few multi-image tasks at limited scale.Source: §2.2 · p. 3
  • MIBench responds with a benchmark spanning three multi-image scenarios and 13 tasks to evaluate MLLM multi-image capabilities comprehensively.Source: §2.2 · p. 3

3 MIBench

MIBench organizes multi-image evaluation into instruction, knowledge-seeking, and in-context-learning scenarios, with tasks covering distinct forms of perception, reasoning, knowledge use, and learning from demonstrations. Its construction uses task-specific question and distractor generation, quality checks, and position-bias-aware evaluation.

3.1 Evaluation Taxonomy

  • MII, MKS, and MIC distinguish cross-image instruction, use of interleaved external knowledge, and answering visual questions from multimodal demonstrations.MKS questions may concern a single image or no visual content; MIC supplies a series of multimodal demos.Source: §3.1 · p. 3

3.1.1 Multi-Image Instruction

  • MII tests general cross-image comparison, subtle differences, visual relations, temporal understanding, and logical or causal reasoning.Source: §3.1.1 · p. 4

3.1.2 Multimodal Knowledge-Seeking

  • MKS tasks cover recognizing query-image objects from reference images, extracting answers from text-rich images, linking query images to textual knowledge, and retrieving visual attributes from interleaved image-text knowledge.Source: §3.1.2 · p. 4

3.1.3 Multimodal In-Context Learning

  • MIC evaluates learning image-label mappings, task patterns, effects on hallucination, and task adaptation from multimodal demonstrations.The tasks use close-ended and open-ended VQA, a POPE-derived hallucination task, and demo-based task learning.Source: §3.1.3 · pp. 4–5

3.2 Data Generation

  • Question forms are generated with GPT-4 prompts and sampled from a question pool, while distractors come from annotations or GPT-4 and MKS knowledge is sampled to match queries.For temporal reasoning, distractors use different temporal relations of the same object; VTK knowledge pairs share a question but have different answers.Source: §3.2 · p. 5

3.3 Quality Control

  • Quality control uses validation or test data, automated checks for visual-input and external-knowledge dependence, and manual review for duplicate or multiple-correct options.Figure 3 presents a qualitative Subtle Difference case in which open-source MLLMs show inferior performance attributed to limited fine-grained perception.Source: §3.3 · p. 5, Figure 3 · p. 6

3.4 Evaluation

  • Multiple-choice accuracy is evaluated while cycling the correct answer across A–D, and a response counts only when the model answers consistently across tests.GPT-4 maps longer responses from early MLLMs to an option when needed.Source: §3.4 · p. 6

4.1 Models

The evaluation covers closed-source models, open-source single-image models, and open-source models that natively accept multiple images. For MIC, it reports only models with multimodal in-context-learning abilities.

  • MII and MKS evaluation groups include closed-source GPT-4V and GPT-4o, open-source single-image models, and open-source models natively supporting multi-image input.The listed native multi-image models are Mantis, Idefics2, and mPLUG-Owl3; open-source models use greedy decoding.Source: §4.1 · p. 6
  • Models lacking multimodal ICL training showed negative ICL effects across four MIC tasks, so results are presented only for OpenFlamingo, MMICL, IDEFICS, and IDEFICS-I.Source: §4.1 · p. 6

4.2 Evaluation Results

Across MII and MKS, closed-source models generally outperform open-source models, with notable weaknesses in fine-grained perception, visual referring, and multimodal knowledge-seeking. MIC results show some image-dependent in-context learning, but abilities vary by task and remain limited.

4.2.1 Multi-Image Instruction & Multimodal Knowledge-Seeking

  • On Temporal Reasoning, MMICL reaches 47.0% accuracy, 21.0% behind GPT-4o; closed-source models outperform open-source models on most MII and MKS tasks.The passage characterizes the open-source models’ lag as significant.Source: §4.2.1 · p. 7
  • Idefics2-I scores 83.1% on General Comparison, 49.7% on Subtle Difference, and 43.9% on Text-Rich Images VQA, while GPT-4V and GPT-4o largely outperform open-source models.The passage attributes the proprietary models’ advantage to their high-resolution input strategy.Source: §4.2.1 · p. 7
  • Multi-image models outperform single-image models on most tasks; mPLUG-Owl3 leads open-source models on most tasks after combining multi-image pre-training and instruction tuning.The reported examples identify interleaved image-text pre-training and multi-image instruction tuning as beneficial.Source: §4.2.1 · p. 7
  • Visual Referring remains difficult because it combines fine-grained perception, spatial correspondence, and relation reasoning; even GPT-4o stays below 50% accuracy.Source: §4.2.1 · p. 7
  • mPLUG-Owl2 scores below 30% on both VTK and TVK and only just over 10% on FVR; Idefics2-I stays below 40% on VTK and TVK.Multi-image models such as Idefics2-I do better than single-image models at using multimodal external knowledge, but the passage reports substantial room for improvement.Source: §4.2.1 · p. 7

4.2.2 Multimodal In-Context Learning

  • On close-ended VQA, removing demo images lowers most models’ performance, with larger declines at more shots, indicating image-label mapping learning from demos.Figure 4 compares accuracy across numbers of demos, including versions with demo images removed.Source: §4.2.2 · p. 7
  • As demo count rises, OpenFlamingo’s open-ended VQA performance changes little without images, whereas IDEFICS and MMICL decline significantly, suggesting different modality reliance.The reported pattern indicates text-driven ICL for OpenFlamingo and some image-dependent ICL for IDEFICS and MMICL.Source: §4.2.2 · pp. 7–8
  • Increasing shots offers limited help against object hallucination, while demo-based task learning varies: IDEFICS gains, OpenFlamingo saturates, and MMICL fails to follow the demo format.Except for MMICL, models can follow the output format with four shots; performance remains well below that with explicit task instructions.Source: §4.2.2 · p. 8

4.3 Analysis

The analyses identify confusion from added images and weaker multi-image reasoning than spatial perception. MKS bottlenecks differ by task: visual perception and reasoning, image comparison, and resolution each constrain performance.

4.3.1 Multi-image Confusion Phenomenon

  • Adding a distractor image lowers performance for three single-image models and slightly lowers Mantis’s, indicating multi-image confusion that can affect MII and MKS.The comparison uses two POPE subsets of 500 samples each: one image versus the same sample with an extra image containing the queried object.Source: §4.3.1 · p. 8

4.3.2 Limited Reasoning Ability

  • When temporal-reasoning distractors use different rather than same objects, mPLUG-Owl2, LLaVA-1.5, and Mantis improve, suggesting their multi-image reasoning is weaker than spatial perception.The passage notes that models can take shortcuts by distinguishing objects in the altered distractor setting.Source: §4.3.2 · p. 8

4.3.3 Bottlenecks of the MKS task

  • For text-linked visual knowledge, removing distractors helps but performance remains poor, pointing to visual perception and reasoning; for vision-linked textual knowledge, the bottleneck is mining information through image comparison.The passage characterizes these as linking by text and linking by image, respectively.Source: §4.3.3 · p. 9
  • On Text-Rich Images VQA, Idefics2-I rises from 46.8% to 59.5% with image splitting, while the broader analysis identifies low resolution as a challenge for rich-text images.Removing distractors also brings some performance boost on this task.Source: §4.3.3 · p. 9

5 Conclusion

MIBench addresses the underexplored evaluation of MLLMs on multi-image inputs with a benchmark spanning three scenarios, 13 tasks, and 13K annotated samples. Its evaluations find significant multi-image challenges despite models’ strength on single-image tasks, and the data is publicly available to support further research.

  • MIBench evaluates multi-image abilities across multi-image instruction, multimodal knowledge-seeking, and multimodal in-context learning, covering 13 tasks and 13K annotated samples.Source: §5 · p. 9
  • Despite strong performance on single-image tasks, existing models face significant challenges with multi-image inputs.Source: §5 · p. 9
  • The annotated benchmark data is publicly available to facilitate further research into multi-image abilities.Source: §5 · p. 9

Limitations

The benchmark’s MII and MKS samples contain two to eight input images because of current MLLMs’ input-length limitations; evaluating larger image counts is left for future work.

  • MII and MKS samples are limited to two to eight input images because of current MLLMs’ input-length limitations.Source: Limitations · p. 9
  • The stated image-count boundary applies to the Multi-Image Instruction and Multimodal Knowledge-Seeking scenarios.Source: Limitations · p. 9
  • The authors identify evaluation over more images as future work, noting that real-world scenarios may involve many images.Source: Limitations · p. 9

A More Details of MIBench

The section clarifies task-specific annotation sampling and the statistics reported for each task, including how MIC image counts and average answer length are defined.

A More Details of MIBench

  • “Sampled from annotations” means task-specific sampling strategies, not simple random sampling from the original annotations.The note points to Section 3.2 for the strategies.Source: A More Details of MIBench · p. 12
  • Table 6 reports each task’s images per sample, average question length, and average answer length.Source: A More Details of MIBench · p. 12
  • MIC’s image number per sample is a range determined by the number of demos; the benchmark totals 125K images, and average answer length averages options for multiple-choice questions and answers for short-answer questions.Source: A More Details of MIBench · p. 12
Loading 2407.15272v2…