PaperlayerView exact revision on arXiv

Source-linked AI summary

MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning

Jiachun Li, Shaoping Huang, Zhuoran Jin, Chenlong Zhang, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

arXiv:2603.02024v1cs.CLcs.AIcs.CV

TL;DR

Existing benchmarks provide limited evidence about how MLLMs reason across diverse real-life scenarios and multi-image inputs. MMR-Life addresses this gap with a curated benchmark spanning seven reasoning types, then evaluates 37 MLLMs and analyzes reasoning paradigms. The results show that real-world reasoning remains challenging, with strong variation across reasoning types and limited gains from thinking modes in open-source models.

  • Problem

    Existing benchmarks do not adequately evaluate MLLM reasoning in everyday real-life scenarios using diverse multi-image inputs.

  • Method

    MMR-Life curates real-life multi-image questions covering seven reasoning types and evaluates both non-thinking and thinking MLLMs alongside reasoning-paradigm analyses.

  • Results

    MMR-Life remains challenging: models perform relatively well on analogical, deductive, and inductive reasoning but struggle with causal, spatial, and temporal reasoning.

  • Takeaways & Limitations

    MMR-Life provides a comprehensive basis for evaluating and analyzing multimodal reasoning across real-life scenarios and for studying more generalizable reasoning systems.

  • Takeaways & Limitations

    The question of whether reinforcement-learning methods on small models overfit specific datasets remains open for future exploration.

Abstract

from arXiv · show

Recent progress in the reasoning capabilities of multimodal large language models (MLLMs) has empowered them to address more complex tasks such as scientific analysis and mathematical reasoning. Despite their promise, MLLMs' reasoning abilities across different scenarios in real life remain largely unexplored and lack standardized benchmarks for evaluation. To address this gap, we introduce MMR-Life, a comprehensive benchmark designed to evaluate the diverse multimodal multi-image reasoning capabilities of MLLMs across real-life scenarios. MMR-Life consists of 2,646 multiple-choice questions based on 19,108 images primarily sourced from real-world contexts, comprehensively covering seven reasoning types: abductive, analogical, causal, deductive, inductive, spatial, and temporal. Unlike existing reasoning benchmarks, MMR-Life does not rely on domain-specific expertise but instead requires models to integrate information across multiple images and apply diverse reasoning abilities. The evaluation of 37 advanced models highlights the substantial challenge posed by MMR-Life. Even top models like GPT-5 achieve only 58% accuracy and display considerable variance in performance across reasoning types. Moreover, we analyze the reasoning paradigms of existing MLLMs, exploring how factors such as thinking length, reasoning method, and reasoning type affect their performance. In summary, MMR-Life establishes a comprehensive foundation for evaluating, analyzing, and improving the next generation of multimodal reasoning systems.

1 INTRODUCTION

MMR-Life addresses the mismatch between existing multimodal benchmarks and everyday reasoning by evaluating MLLMs on diverse multi-image, real-life scenarios. Results across 37 models show that this setting remains difficult and exposes substantial differences among reasoning types.

  • Motivation: Existing benchmarks often use expert knowledge, synthetic puzzles, or limited reasoning types that diverge from everyday multimodal reasoning.Existing multi-image benchmarks also include non-reasoning tasks or focus on a narrow reasoning type.
  • Benchmark proposal: MMR-Life evaluates multimodal multi-image reasoning across real-life scenarios without requiring domain-specific expertise.Questions require models to extract information from multiple real-life images and derive new conclusions.
  • Benchmark proposal: 2,646 questions cover 7 reasoning types, including abductive, analogical, causal, deductive, inductive, spatial, and temporal reasoning.The benchmark is designed to encompass reasoning abilities necessary for everyday situations.
  • Findings: 58.69% accuracy is achieved by GPT-5, while Gemini-2.5-Pro reaches 56.86%, and both remain 14% below human performance.These results come from extensive evaluation on 37 advanced MLLMs.
  • Findings: Analogical, deductive, and inductive reasoning are comparatively stronger, whereas causal, spatial, and temporal reasoning remain bottlenecks.The benchmark also reveals higher-order reasoning clusters, such as analogical and inductive reasoning.

2 THE MMR-LIFE BENCHMARK

MMR-Life is a curated benchmark of real-life multi-image questions designed to test seven reasoning types without relying on domain-specific expertise. Its construction combines diverse image collection, task-specific question generation, multiple-choice distractors, and quality filtering.

  • Overview: MMR-Life contains 2,646 multiple-choice questions based on 19,108 images, covering 7 reasoning types and 21 tasks.Images are predominantly sourced from real-life contexts such as domestic life, dining, and sports.
  • Data collection: The data-collection pipeline combines public image datasets, open-web sources, and frames from existing multi-image or video reasoning benchmarks.Low-quality images are removed, with detailed protocols reported separately.
  • Task design: Seven task categories cover abductive, analogical, causal, deductive, inductive, spatial, and temporal reasoning in everyday situations.The categories map observations, similarities, causes, rules, and other image-based evidence to conclusions.
  • Question generation: Question-answer pairs are generated automatically or manually according to task requirements, including heuristic synthesis when image sequences provide sufficient information.Temporal sequence prediction can be generated from sequential information already present in the images.
  • Question generation: Five-option multiple-choice questions accommodate tasks without a single obvious answer, while heuristic rules generate incorrect image or text options.Temporal distractors can be sampled from frames preceding the input sequence.
  • Quality control: Three filtering steps remove questions that smaller models answer too easily and check generated-answer format and quality.Difficulty filtering uses Qwen2.5-VL-7B, Gemma3-4B, and InternVL3.5-8B.
  • Benchmark comparison: Unlike many existing benchmarks, MMR-Life excludes symbolic images and uses newly annotated questions rather than directly sampling existing sources.This design targets natural images encountered in daily life and reduces contamination risk.

3 MAIN EXPERIMENT

The main experiments evaluate non-thinking and thinking MLLMs, using standardized zero-shot CoT evaluation and human comparison. MMR-Life remains difficult: models show weak overall and spatial reasoning performance, while open-source thinking provides limited improvement.

  • 37 advanced MLLMs are evaluated on MMR-Life alongside human performance.The experiments include non-thinking and thinking models, with repeated runs for open-source models and a human mini-test set.
  • 58.69% is GPT-5's accuracy on MMR-Life, 14% below human performance.This contrasts with GPT-5's nearly 90% accuracy on GPQA and MMMU reported in the passage.
  • 25.10% is the highest model accuracy for spatial reasoning, versus 79.76% human accuracy.Models perform relatively well on analogical, deductive, and inductive reasoning but struggle with causal, spatial, and temporal reasoning.
  • 29.01% versus 27.15% is the average accuracy of open-source no-thinking versus thinking models.The thinking mode improves closed-source models generally but does not improve open-source models on average.

4 THINKING PATTERN ANALYSIS

The analysis examines how thinking length and enhancement methods relate to performance across reasoning types and model scales. Longer reasoning is not uniformly beneficial, and enhancement gains diminish or reverse as models become larger.

  • Thinking length: Reasoning performance scales roughly with the logarithm of reasoning length, but some open-source thinking models combine long outputs with low accuracy.MiMo-VL-7B-RL and QVQ-72B-Preview occupy the lower-right region of the response-token versus accuracy plot.
  • Thinking length: Longer CoT improves analogical reasoning but can worsen inductive reasoning.For inductive reasoning, CoT hurts no-thinking models and larger reasoning budgets do not improve thinking models.
  • Enhancement methods: Across model scales from 7B to 72B, enhancement methods increasingly shift from gains to drops relative to CoT.On Qwen-2.5-VL-72B, BoN and GRPO perform worse than simply applying CoT.
  • Enhancement methods: GRPO generalizes worse than BoN on all three evaluated models.The authors leave open whether RL methods on small models merely overfit specific datasets.
  • Reasoning-type relationships: 0.97 versus 0.40 are reported Pearson correlations for inductive–analogical and spatial–inductive reasoning.Hierarchical clustering also forms groups such as Ana–Ind, suggesting higher-order reasoning patterns in MLLMs.

5 ERROR ANALYSIS

Error analysis of GPT-5 and Gemini-2.5-Pro identifies reasoning mistakes as the largest error category, alongside abstraction, knowledge, and perception failures.

  • 32% of the sampled errors are reasoning errors, including causal inversion, temporal confusion, and missing key steps.Temporal confusion accounts for 42%, while causal inversion and missing key steps each account for 24% of reasoning errors.
  • 17% of errors are abstraction errors, and knowledge and perception errors account for 17% and 12%, respectively.These failures involve associations, recalling knowledge, and identifying static attributes or dynamic changes.

6 RELATED WORK

Related work spans multimodal reasoning-enhancement methods and benchmarks that combine reasoning with expert knowledge or symbolic patterns. MMR-Life is positioned within this literature as a real-life multimodal reasoning evaluation benchmark.

  • Multimodal Reasoning Enhancement Methods: CoT has been extended to MLLMs through prompt-guided methods including IPVR, CCoT, and VisualSketchpad.These approaches combine reasoning with perception to improve multimodal reasoning reliability.
  • Multimodal Reasoning Enhancement Methods: VL-Rethinker, MM-Eureka, and MiMo-VL are multimodal reasoning models evaluated by the benchmark.The benchmark is intended to guide further optimization of MLLM reasoning methods and models.
  • Multimodal Reasoning Benchmarks: Existing multimodal benchmarks assess reasoning using world knowledge across STEM fields or symbolic patterns.Examples include GPQA, OlympiadBench, MME-CoT, MMR-V, MMLU-Reason, and related symbolic-pattern benchmarks.

7 CONCLUSION

MMR-Life evaluates multimodal reasoning across seven reasoning types using multiple real-life images, while emphasizing commonsense solvability and natural-image inputs. The benchmark also analyzes how input format and model reasoning paradigms relate to reasoning ability.

  • Conclusion: MMR-Life evaluates MLLMs across seven reasoning types using multiple real-life images.The benchmark is designed to assess multimodal reasoning abilities across varied real-life scenarios.
  • Conclusion: MMR-Life further analyzes relationships among thinking length, enhancement methods, and MLLM reasoning abilities.The authors present this analysis as groundwork for developing more generalizable AI systems.
  • Design principles: Real-life reasoning is defined as applying diverse reasoning capabilities to everyday problems represented by multiple natural images and textual descriptions.Purely abstract diagrams or symbolic renderings are excluded.
  • Design principles: Questions are designed to be solvable through basic human commonsense reasoning and general logic rather than complex domain-specific knowledge.This criterion distinguishes MMR-Life from benchmarks that may require specialized expertise.
  • Input format: The benchmark uses multiple images instead of continuous video because video offers limited coverage of parallel reasoning types, lower data diversity, and potentially noisy sampled frames.The design aims to assess reasoning directly while minimizing interference from irrelevant frames.

C.1 DATA SOURCES OF DIFFERENT TASKS

MMR-Life sources natural images while enforcing licensing, annotation, and task-format constraints. Annotators require multiple-image, English, unambiguous questions that can be solved without complex domain knowledge.

  • Data sources: Data collection follows copyright and licensing regulations and excludes symbolic diagrams and other non-photographic forms.The dataset is restricted to natural images in accordance with the benchmark definition.
  • Annotation guidelines: Every question must contain at least two images, be written in English, and have one unambiguous answer option.These requirements standardize the annotation format.
  • Annotation guidelines: Questions must be solvable without complex domain-specific knowledge and conform clearly to their designated reasoning types.The guidelines aim to distinguish tasks across reasoning categories.
  • Dataset diversity: The benchmark documents source and image-type diversity through task-level source tables and visualizations of image types and input-image counts.Additional figures illustrate the range of included task types.

E TASK DETAILS

MMR-Life contains tasks spanning behavioral, causal, analogical, inductive, spatial, and temporal reasoning, alongside physical, combinatorial, rule-based, and planning problems. Each task requires models to infer relationships, sequences, outcomes, or choices from multiple images.

  • Abductive and causal reasoning: Behavioral and causal tasks infer motivations, interaction causes, and root events in character or physical collision scenarios.The tasks use environmental clues, behavior cues, situational factors, and collision chains.
  • Analogical reasoning: Analogical tasks transfer visual relations between animals, product styles, and artistic works to select matching candidates.The model must identify shared relational or stylistic patterns across image sets.
  • Prediction and structured reasoning: Outcome-prediction tasks model character reactions, collision propagation, fluid changes after barrier removal, and product quantities under material constraints.These tasks combine contextual, physical, counterfactual, or combinatorial reasoning.
  • Deductive and inductive reasoning: Rule-based and inductive tasks determine poker winners, reconstruct cooking order, predict bird distributions, recognize plant diseases, and extend sports patterns.The required reasoning ranges from explicit game rules to learned visual or temporal regularities.
  • Spatial reasoning: Spatial tasks infer directions, estimate camera rotations, and plan robot paths using constrained turns and obstacle avoidance.The tasks test both relational scene understanding and action-sequence planning.
  • Temporal reasoning: Temporal tasks order crowd activities, predict the next driving frame, and locate when an activity occurs within a video sequence.Models integrate positions, actions, road geometry, traffic, and environmental cues.

F.1 DETAILED EXPERIMENTAL SETUP

The experiments evaluate a broad set of multimodal language models under fixed inference settings and documented prompting procedures. Full and mini-test-set comparisons are reported separately.

  • Models: The evaluation includes 37 multimodal language models with corresponding model versions listed for reproducibility.The listed systems include models from OpenAI, Anthropic, and Google.
  • Parameters: Inference uses temperature 0.5, top p 0.5, and seed 17.These fixed parameters define the reported model-inference setup.
  • Prompts: The main-experiment prompt is illustrated in Figure 18.The prompt specification is referenced rather than reproduced in the passage.
  • Results reporting: Full evaluation results for 37 MLLMs are presented in Table 6.This table provides the comprehensive model comparison.
  • Results reporting: Performance comparisons on the mini test set are presented separately in Table 7.The mini test set serves as an additional evaluation report.

G DETAILS OF THINKING PATTERN ANALYSIS

The thinking-pattern analysis compares base and reinforcement-learning model setups across multiple model families. These selections are intended to examine whether conclusions generalize across training datasets.

  • The base setting includes MiMo-VL-7B-SFT, RL-Onevision, Qwen-2.5-VL-32B, and Qwen-2.5-VL-7B with Chain-of-Thought prompting.
  • The reinforcement-learning setup uses MiMo-VL-7B-RL, RL-Onevision-RL, MM-Eureka-32B, and VL-Rethinker-7B.
  • The selected models are trained on various datasets to illustrate the generalizability of the analysis conclusions.

H CASE STUDY

The case-study materials provide correct and incorrect examples across several MMR-Life reasoning tasks, alongside full and mini-set performance comparison tables. They cover abductive, analogical, causal, spatial, and temporal task instances.

  • Case studies include correct and incorrect responses by GPT-5 and Gemini-2.5-Pro.
  • Tables 6 and 7 provide full and mini-set performance comparisons for state-of-the-art multimodal large language models on MMR-Life.
  • The examples cover abductive tasks such as Human Activity Attribution, Character Interaction Attribution, and Multi-Hop Collision Attribution.
  • Analogical examples address Animal Relation Inference, Product Similarity Inference, and Artwork Style Inference.
  • Causal examples ask models to predict collision, fluid-flow, and character-interaction consequences from image-based events.
Loading 2603.02024v1…