MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

1 Introduction

• Single images capture only a particular angle and moment, whereas multiple images provide perspectives and time points for richer scene understanding.The introduction gives action prediction from consecutive images and 3D navigation from multi-view images as examples.Source: §1 · p. 1
• Earlier evaluations lagged model development: single-image benchmarks miss multi-image complexity, while recent multi-image benchmarks cover few relationships or task types.The introduction notes that Video-MME focuses on temporal relationships and MUIRBENCH omits spatial relationships.Source: §1 · p. 2
• MMIU covers 7 relationship types, 52 tasks, 77k images, and 11.6k questions, with broader coverage than prior benchmarks.Its analytical tools include relationship-level comparison, task-map discovery of in- and out-of-domain tasks, and SFT-based task-difficulty analysis.Source: §1 · p. 3, Table 1 · p. 4
• GPT-4o reaches only 55.7% accuracy on MMIU, while models perform better on semantic than temporal or spatial relationships.The reported findings also identify difficulty with out-of-domain 3D perception and temporal reasoning tasks.Source: §1 · p. 3
• MMIU’s analyses aim to expose current models’ limitations and performance gaps and inform future model and data improvement.The paper identifies spatial understanding as a particularly challenging area for current LVLMs.Source: §1 · pp. 3–4

2.1 Large Vision-Language Models

• Multi-image LVLM development has used interleaved image-text pretraining and extensive multi-image instruction tuning, but evaluation across broad task ranges remained insufficient.Examples of interleaved corpora and resulting models include MMC4, Omnicorpus, Deepseek-VL, and Idefics; Mantis and LLaVA-Next-interleave exemplify instruction tuning.Source: §2.1 · p. 4

2.2 Large Vision-Language Models Benchmarks

• Most LVLM benchmarks target single-image scenarios, while multi-image benchmarks often have limited capability or task coverage.MANTIS-Eval emphasizes size perception, Video-MME emphasizes temporal sequences, and MILEBENCH lacks tasks including 3D spatial understanding and low-level semantics.Source: §2.2 · pp. 4–5

3.1 Benchmark Overview

• MMIU combines 77,659 images and 11,698 questions across 7 relationships and 52 tasks, including 19 tasks with 40 unanswerable questions each.The benchmark averages 6.64 images per instance and offers relationship comparisons, task-map analysis, and SFT-based task-difficulty analysis.Source: §3.1 · p. 5

3.2 Data Curation Process

• Multiple-choice questions use metadata-derived options generated with manually designed rules or GPT-4o, and include answerable and unanswerable samples.For example, GPT-4o generates plausible incorrect options for 3D question answering, while image retrieval uses randomly selected incorrect images.Source: Figure 2 · p. 6, §3.2 · p. 7
• MMIU organizes evaluation by enumerating semantic, temporal, and spatial image relationships, then assigning multiple multi-image tasks to each relationship.The relationships are refined into low- and high-level semantic, continuous and discrete temporal, and 2D and 3D spatial types.Source: §3.2 · p. 6
• The curation process searches for task-relevant datasets, standardizes samples as metadata, and limits most tasks to 200 randomly selected samples.Metadata records task description, question, answer, input context, and images; the team manually checks accuracy and convertibility to multiple-choice format.Source: §3.2 · p. 7
• Building the benchmark required careful question design and answer verification, particularly for 3D spatial tasks involving relative camera poses.For 3D pose estimation, relative poses are derived by matrix multiplication and checked against image-pair correspondences.Source: §3.2 · p. 7

4.2 Main Results

• GPT-4o leads at 55.7% average accuracy, while InternVL2 leads open-source models at 50.3%, a 5.4% accuracy gap between closed- and open-source models.Table 3 reports accuracy across 24 models and 52 tasks, with Overall computed across tasks.Source: Table 3 · p. 8, §4.2 · p. 9
• Multi-image fine-tuning improved performance: Mantis gained 17.8% accuracy over Idefics2, while single-image-trained GLM4V reached 37.4% and surpassed some multi-image models.The authors also identify single-image understanding as a foundation for multi-image comprehension.Source: §4.2 · p. 9

4.3.1 Performance across Image Relationships

• Models perform best on semantic relationships, moderately on temporal tasks, and worst on spatial relationships; GPT-4o scores 28% on temporal ordering and 21.5% on visual ordering.Models struggle with both 2D and 3D positional relations and with high-level semantic reasoning, retrieval, and reasoning-intensive temporal tasks.Source: §4.3.1 · pp. 9–10, Figure 3 · p. 10

4.3.2 Analysis on the Task Map

• Recognition and captioning tasks tend to be in-domain, while temporal ordering and 3D spatial reasoning are out-of-domain and difficult for most models.Recognition or captioning tasks require less comparison and reasoning between images; the authors suggest limited 3D vision-language training data may contribute to weak 3D-task performance.Source: §4.3.2 · pp. 10–11, Figure 4 · p. 11

4.3.3 Task Learning Difficulty

• AccSFT and AccModel task accuracies have a Spearman correlation of 0.66; ordering, retrieval, and many-image tasks remain difficult and may require stronger memory and long-context capacity.AccSFT reflects how difficult a task is to fit after fine-tuning, while AccModel reflects difficulty for current models.Source: §4.3.3 · pp. 11–12, Figure 5 · p. 11

4.4 Ablation Study

• On high-performing tasks, unanswerable questions reduced GPT-4o and InternVL1.5 accuracy, but accuracy changes showed no significant pattern on already difficult tasks.Figure 6 compares answerable and unanswerable questions for the two representative models, with a red line marking each model’s average accuracy.Source: Figure 6 · p. 12, §4.4 · p. 12
• For single-image models, concatenating visual tokens did not outperform image concatenation; the image method did significantly better for the LLaVA series, while GLM-4V was relatively consistent.Figure 7 compares the two testing methods across tasks.Source: §4.4 · pp. 12–13, Figure 7 · p. 13

5 Conclusion

• Current LVLMs, including GPT-4, struggle with complex multi-image tasks, according to evaluation of 24 models on MMIU.The benchmark includes seven image-relationship types and 52 tasks.Source: §5 · p. 13

A.1 Multi-image Relations

• Semantic relationships cover low-level visual features, objective objects and attributes, and subjective themes or emotions.Examples include lighting, image quality, object relationships, thematic association, and emotional association.Source: §A.1 · p. 22
• Temporal relationships cover discrete event sequences and continuous video-frame perception or reasoning; spatial relationships cover 2D transformations and 3D viewpoint or depth variation.The three broad relationship groups are semantic, temporal, and spatial.Source: §A.1 · p. 22

A.2 Hierarchical Structure of MMIU

• MMIU organizes seven image-relationship types into 52 tasks, with task descriptions and data sources catalogued across temporal, spatial, and semantic relationships.The benchmark presents task distributions by relationship type and task-level descriptions and sources.Source: §A.2 · pp. 22–23, Table 4 · p. 22, Table 5 · p. 24
• Its spatial task examples span 3D object detection and pose estimation, scene reconstruction, object tracking, instance segmentation, and depth estimation.These tasks use multiple views, image sequences, or RGB-D inputs to produce spatial estimates or outputs.Source: §A.2 · p. 27

A.3 Task Abbreviations

• The paper uses abbreviations to streamline discussion of its many tasks and evaluated models.Table 8 lists the abbreviations and their full terms.Source: §A.3 · p. 31, Table 8 · p. 31

A.4 Construction

• Task metadata supports conversion into multiple-choice questions without losing information.The metadata format is referenced in Table A.4.Source: §A.4 · p. 31
• The construction procedure considers five minimal-change strategies for turning answerable instances into unanswerable ones.Table 9 summarizes causes of unanswerability, and a task can correspond to multiple causes.Source: §A.4 · p. 31, Table 9 · p. 32

B.1 Model Details

• The study evaluates 24 LVLMs and lists each model’s parameter size, visual encoder, and language model architecture.The evaluation follows the OpenCompass protocol.Source: §B.1 · p. 33, Table 10 · p. 33

B.2 Model Prompts

• Prompts combine a question, answer options, an answer-format hint, and task context, with images inserted into the text.The example answer-format hint requests a direct option such as A, B, C, or D.Source: §B.2 · p. 33

Model Prompts

• The prompt template contains context, question, four answer choices, and a hint to answer directly with an option.The template appears in the model-prompts appendix.Source: Model Prompts · p. 34

B.3 Multitask Analysis

• Figure 8 compares the average performance of 24 LVLMs across seven specific image relationships.The cited analysis directs readers to the main text for detailed interpretation.Source: §B.3 · p. 33, Figure 8 · p. 34

• The analysis examines error cases from GPT-4o, Claude3.5-Sonnet, and InternVL2-Pro, the three best-performing models.Source: §B.4 · p. 34
• One error case is visualized for each type of image relationship across Figures 9–15.Source: §B.4 · p. 34
• For lengthy model responses or questions, the analysis mostly extracts their important parts for display.Source: §B.4 · p. 34

B.5 Ablation Study

• The answerability experiment tests representative LVLMs on 19 tasks with both answerable and unanswerable questions.Source: §B.5 · p. 34
• The input-method experiment compares concatenated visual tokens with concatenated images for single-image models on multi-image tasks.Source: §B.5 · p. 34
• Table 11 summarizes accuracy for LVLMs across 21 tasks and computes the overall score across all tasks.Source: Table 11 · p. 35

C Task Map

• The task map combines task similarity with model performance to analyze strengths and weaknesses while avoiding reliance on meta-task analysis alone.Source: C Task Map · p. 35

C.1 Construction

• The authors fine-tune QwenVL-chat on multiple-choice samples from 52 tasks and define each task vector as the weight change from the initial model.Source: §C.1 · p. 36
• For most tasks, fine-tuning runs for 20 epochs, extending to 60 for some tasks with low initial accuracy; LoRA reduces vector length from 9.6B to 3.5M.Source: §C.1 · p. 36
• Cosine distances between task vectors form a 52x52 task map, where related tasks such as GNAP and GuAR, and VO, TO, and VCo, cluster together.Source: §C.1 · p. 36

C.2 Analysis

• Most models perform worse than random selection on multi-image tasks overall, but do relatively well on recognition or captioning tasks in clusters 7, 8, and some of cluster 2.Source: §C.2 · p. 37
• These multi-image tasks focus on overall image perception, requiring less comparison and reasoning between images; the model has also demonstrated strong high-level perception on single images.Source: §C.2 · p. 38
• Models perform poorly on out-of-domain tasks in clusters 4, 5, and 6, including semantic or sequential relationships and 3D spatial tasks.Source: §C.2 · p. 38
• Clusters 4 and 6 involve semantic relationships or sequential ordering, while cluster 5 covers 3D spatial tasks such as detection and tracking.Source: §C.2 · p. 38