Source-linked AI summary
More Images, More Problems? A Controlled Analysis of VLM Failure Modes
Anurag Das, Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Bernt Schiele, Georgios Tzimiropoulos, Brais Martinez
TL;DR
Multi-image LVLM capabilities lack comprehensive analysis of their weaknesses and causes, especially for cross-image aggregation and multi-concept tracking. The paper introduces MIMIC for controlled diagnosis and proposes synthetic multi-image fine-tuning plus attention masking, achieving substantial gains across scenarios and existing benchmarks. The study’s controlled MS-COCO construction and focus on semantic reasoning leave specialized domains and pixel-perfect small-detail perception outside scope.
Problem
Comprehensive evidence about LVLM weaknesses and their causes in multi-image settings remains lacking despite existing benchmarks and growing model capabilities.
Method
The paper introduces MIMIC and combines procedurally generated multi-image supervision with layer-wise attention analysis and tailored attention masking.
Results
The proposed fine-tuning strategies substantially improve multi-image performance, including cross-image aggregation, and establish new state-of-the-art results on existing benchmarks.
Takeaways & Limitations
Controlled diagnosis paired with data- and optimization-centric fine-tuning provides a basis for improving multi-image LVLM understanding across evaluated scenarios.
Takeaways & Limitations
MIMIC is constructed from MS-COCO, so extending its controlled methodology to specialized domains remains outside the demonstrated scope.
Abstract
from arXiv · showhide
Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored. While existing benchmarks have initiated the evaluation of multi-image models, a comprehensive analysis of their core weaknesses and their causes is still lacking. In this work, we introduce MIMIC (Multi-Image Model Insights and Challenges), a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs. Using MIMIC, we conduct a series of diagnostic experiments that reveal pervasive issues: LVLMs often fail to aggregate information across images and struggle to track or attend to multiple concepts simultaneously. To address these failures, we propose two novel complementary remedies. On the data side, we present a procedural data-generation strategy that composes single-image annotations into rich, targeted multi-image training examples. On the optimization side, we analyze layer-wise attention patterns and derive an attention-masking scheme tailored for multi-image inputs. Experiments substantially improved cross-image aggregation, while also enhancing performance on existing multi-image benchmarks, outperforming prior state of the art across tasks. Data and code will be made available at https://github.com/anurag-198/MIMIC.
1 Introduction
This work addresses the lack of comprehensive analysis of LVLM weaknesses in multi-image settings by introducing MIMIC and complementary data- and optimization-centric remedies. The study finds failures in cross-image aggregation, multi-concept tracking, and distractor handling, while reporting substantial gains from the proposed fine-tuning strategies.
- Benchmark and motivation: MIMIC provides a controlled benchmark for systematically evaluating multi-image LVLM capabilities and limitations.It procedurally generates multi-image sequences with controlled information spread, distractors, object-instance distributions, sequence length, and query complexity.
- Diagnostic findings: The study identifies failures in aggregating information across images, tracking multiple concepts, and resisting visual distractors.The analysis attributes these shortcomings to sequence-modeling limitations, training-data biases, poor inter-image communication from causal attention, and multi-image reasoning complexity.
- Proposed remedies: The paper introduces targeted synthetic multi-image supervision and attention masking derived from layer-wise attention analysis.The data-centric strategy composes rich training examples, while the optimization-centric strategy tailors attention behavior to multi-image inputs.
- Results: The proposed fine-tuning strategies substantially enhance performance across multi-image scenarios and establish new state-of-the-art results on existing benchmarks.The contributions combine both methods and report improvements in multi-image contexts.
2 Related work
Prior work expanded LVLM evaluation and multi-image support, but existing benchmarks and analyses leave controlled, granular diagnosis of multi-image reasoning limitations incomplete. This work instead probes information distribution, query complexity, distractors, and internal mechanisms while pairing the analysis with data- and optimization-level solutions.
- Multi-image LVLMs: LVLM research progressed from single-image understanding toward multi-image and video inputs through architectural changes and multi-image instruction tuning.Recent systems support reasoning across multiple visual inputs, while video is represented as sequences of extracted frames.
- Evaluation of LVLMs: Existing benchmarks evaluate increasingly broad skills, but many earlier efforts focus on single-image understanding or limited question diversity.Later benchmarks broaden task coverage and reasoning demands, while multi-image benchmarks such as MuirBench and Blink specifically assess multi-image understanding.
- Benchmark novelty: MIMIC provides more granular control over information distribution, query complexity, and distractor presence than prior multi-image evaluations.Its task design enables selective exploration of performance across controlled dimensions rather than repurposing existing datasets alone.
- Analysis of LVLMs: Prior multi-image analyses often emphasize retrieval or localization and do not fully control confounding factors or identify root causes beyond data scarcity.MIMIC combines controlled probing with internal-behavior analysis and proposed remedies at both data and optimization levels.
3 Challenges and Insights in Multi-Image LVLMs
MIMIC is a controlled benchmark for diagnosing multi-image LVLM behavior across information distribution, query complexity, distractors, scaling, and multi-concept tracking. Experiments show that models struggle with long visual sequences, cross-image aggregation, distractors, and simultaneous concept tracking.
- Benchmark design: MIMIC generates controlled multi-image sequences from MS-COCO annotations to vary information spread, distractors, object distributions, and sequence length.The benchmark supports decorrelated, fine-grained analysis across six complementary dimensions.
- Benchmark design: MIMIC evaluates Counting, Listing, Common, and Odd-One tasks using open-ended questions designed to probe distinct aspects of multi-image reasoning.Counting varies object classes, information spread, distractors, and total image count; Listing tests exhaustive extraction and aggregation.
- Sequence-length effects: 4−8× sequence-length reduction via 1-D pooling significantly improves performance, while pixel-space information reduction does not, implicating long-sequence understanding.The control experiment keeps sequence length fixed while reducing pixel information, separating sequence-length effects from information loss.
- Sequence-length effects: LVLM performance peaks when vision-token length matches roughly one or two images, indicating primarily single-image behavior and limited practical multi-image integration.For LLaVA-OV, the peak occurs near the token count produced by one or two 384×384 images.
- Observed failure modes: 79.0% falls to 66.5% with 1 versus 34 distractors for one query image, while two query images fall from 75.0% to 12.5%.The sharper decline when information is spread across images shows that distractors compound aggregation difficulty.
- Observed failure modes: Inter-image attention is strong in early layers but becomes predominantly intra-image in deeper layers, marking a shift from cross-image integration to individual-image focus.The transition occurs around the middle of the network and may contribute to aggregation difficulty.
4 Method
The method combines synthetic multi-image fine-tuning with layer-wise attention masking. These complementary strategies target the data and optimization factors identified in the diagnostic analysis.
- Fine-tuning strategies: The authors investigate two complementary fine-tuning approaches: synthetically generated multi-image data and optimization-centric attention masking.The strategies are derived from the failure modes identified through zero-shot MIMIC evaluation.
- Attention masking: Attention masking restricts vision tokens to attend only to tokens from the same image in selected layers while leaving text-token attention unchanged.The design is motivated by the observed reduction of inter-image attention in deeper layers.
5 Results
The proposed methods improve multi-image performance across MIMIC and existing benchmarks, with gains linked to cross-image aggregation, multi-concept reasoning, and efficient attention control.
- 41.7 to 51.3%: masked attention improves the 7B model’s overall MuirBench score over the LLaVA-OV baseline.
- 26.4 to 49.4 and 54.0 to 63.8: MIMIC average scores improve for the 0.5B and 7B models, respectively.The largest gains occur on Common and Odd-One, which probe information aggregation and multi-concept reasoning.
- Cross-task generalization: Training on Common generalizes to Counting and Listing but not Odd-One, while Listing improves all other tasks and Counting primarily benefits Odd-One.Common aggregates information across images, whereas Odd-One localizes distinctive evidence within a single image.
- Efficiency and attention analysis: ∼81%: masked finetuning reduces FLOPs on the 0.5B backbone while outperforming full finetuning.The results support selectively constraining inter-image attention as an efficient strategy.
- Efficiency and attention analysis: Masking deeper layers 12–23 performs best, whereas masking early layers significantly degrades accuracy.The ablation suggests that early layers are important for cross-image information aggregation.
- Qualitative analysis: The qualitative attention example shows the baseline missing the relevant object in the third image while the proposed method attends across all images and counts correctly.The visualization is taken at the LLM’s 15th layer.
6 Conclusions
MIMIC exposes a single-image behavioral pattern in current state-of-the-art LVLMs, and the proposed training strategies address key multi-image failure modes while achieving new state-of-the-art results.
- Current state-of-the-art LVLMs struggle to aggregate information across inputs and track multiple concepts amid visual distractors.
- The paper introduces synthetic data-centric fine-tuning and optimization-centric attention masking to address these multi-image failures.
- The contributions establish new state-of-the-art results and a foundation for future research in multi-image understanding.
7 Limitations
The study’s conclusions are bounded by its controlled benchmark domain, sequence-resolution trade-offs, and focus on open-weight architectures.
- Benchmark domain: MIMIC is built from MS-COCO, so extending its controlled methodology to dense documents or medical imaging remains future work.The MS-COCO construction controls confounders such as object counts and occlusion levels.
- Resolution trade-offs: Reducing sequence length improves multi-image reasoning, but pixel-perfect tasks involving extremely small details may require adaptive resolution strategies.Adaptive resolution was outside the study’s scope.
- Architectural scope: The analysis focuses on open-weight models, while additional validation may be useful for reinforcing conclusions about closed models.
A.1 Additional analysis
Additional analyses test whether multi-image failures arise from multiple images themselves, examine attention behavior and efficiency, and evaluate fine-tuning across image counts and benchmarks.
- Stitching experiment: Stitching controls for vision-token count by combining multiple images into one grid composite, isolating whether failures stem from multiple images.The experiment keeps the total number of vision tokens similar between stitched and multi-image inputs.
- Efficiency analysis: Full self-attention costs O((Nt + Nv)^2d + (Nt + Nv)d^2), whereas masked attention reduces visual-token interactions by restricting them within image blocks.The standard expression separates self-attention computation from the MLP term.
- Masked attention: Masked attention restricts visual tokens to their own image blocks while preserving global visibility for text tokens.This creates a block-diagonal visual-attention pattern and is described as computationally efficient.
- Performance versus image count: Across Listing, Odd-One, and common tasks, performance is analyzed as a function of the total number of input images.The analysis extends beyond counting to assess scaling behavior across additional MIMIC tasks.
- Layer-wise attention: In deeper layers, attention becomes mostly intra-image after substantial early-layer inter-image attention, and this pattern holds for both four and six images.The authors interpret the repeated pattern as intrinsic to the model rather than dependent on the number of input images.
- Counting and benchmark comparisons: 45.8% versus 9% accuracy is observed when four object instances are distributed across four images after fine-tuning, indicating improved cross-image aggregation.Similar gains are reported across different instance distributions; the fine-tuned model also improves 3.4% over the baseline in an extended comparison.
A.2 Implementation Details.
The implementation combines synthetic MIMIC supervision with original multi-image instruction data, while separately applying full fine-tuning or LoRA-based masked attention.
- Training data: The unified training set combines synthetic MIMIC samples with approximately 580K original LLaVA-OV multi-image instruction-tuning samples.The synthetic data is generated through the MIMIC pipeline.
- Training data: Synthetic MIMIC training data provides explicit cross-image supervision, supports multi-turn conversations and option-based responses, and contains sequences of up to 10 images.Approximately 50K samples are included from each MIMIC subtask.
- Fine-tuning strategies: The optimization-centric strategy applies LoRA to language-model layers and restricts vision tokens to attend within the same image, while the data-centric strategy fully fine-tunes without masking.The two strategies therefore differ in both parameter adaptation and attention constraints.
- Optimization settings: Training uses 8 NVIDIA H100 GPUs with approximately 80GB memory each, and the attention-masking strategy uses LoRA rank 128.These settings apply to the reported implementation.
A.3 Additional MIMIC details
The appendix provides dataset statistics, prompt-construction details, attention visualizations, scaling analysis, training-format examples, and benchmark samples across MIMIC task categories.
- Dataset details: Table 9 reports statistics for the synthetic MIMIC training dataset based on OpenImagesv7.The dataset is distinct from the evaluation benchmark in its source and training-data construction.
- Prompt templates: Each task prompt combines one randomly sampled task-specific template with one randomly sampled connector template: P = Ptask∥Pconnector.The resulting prompt is formed by concatenating the two sampled templates.
- Attention analysis: Figure 11 visualizes inter-image and intra-image token attention across layers for six images.The figure complements the layer-wise attention analysis of multi-image inputs.
- Counting analysis: Figure 10 examines unbalanced counting performance as the number of query images and total images vary for LLaVA-OV 72B, Qwen2.5-7B, and Qwen3VL-8B.The comparison covers both larger and more recent models.
- Training examples: Figure 11 shows MIMIC training examples in LLaVA format with multi-turn conversations and option-based answers.These examples illustrate the format of the synthetic training data.
- Evaluation examples: Figures 13–16 provide MIMIC evaluation samples from the common, Odd One, Listing, and Counting categories.The figures show examples spanning all four named benchmark categories.