PaperlayerView exact revision on arXiv

Source-linked AI summary

MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning

Jiachun Li, Shaoping Huang, Zhuoran Jin, Chenlong Zhang, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

arXiv:2603.02024v1cs.CLcs.AIcs.CV

TL;DR

Existing benchmarks do not fully reflect everyday reasoning or cover diverse reasoning types across multiple images. MMR-Life evaluates MLLMs on real-life multi-image questions and finds substantial performance disparities across reasoning types, with GPT-5 achieving 58.69% accuracy.

  • Problem

    Existing benchmarks often use tasks unlike everyday reasoning and lack multi-image inputs spanning diverse reasoning types.

    Source: §1 · p. 2

  • Method

    MMR-Life evaluates MLLMs with questions requiring integration of multiple real-life images across seven reasoning types, without domain-specific expertise.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Models show substantial performance disparities across reasoning types, with bottlenecks in causal, spatial, and temporal reasoning; GPT-5 achieves 58.69% accuracy on MMR-Life.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMR-Life’s analysis of thinking length, enhancement methods, and reasoning abilities lays a foundation for developing more generalizable AI systems.

    Source: §7 · p. 10

    Whether reinforcement-learning methods on small models overfit to specific datasets remains open for future work.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Recent progress in the reasoning capabilities of multimodal large language models (MLLMs) has empowered them to address more complex tasks such as scientific analysis and mathematical reasoning. Despite their promise, MLLMs' reasoning abilities across different scenarios in real life remain largely unexplored and lack standardized benchmarks for evaluation. To address this gap, we introduce MMR-Life, a comprehensive benchmark designed to evaluate the diverse multimodal multi-image reasoning capabilities of MLLMs across real-life scenarios. MMR-Life consists of 2,646 multiple-choice questions based on 19,108 images primarily sourced from real-world contexts, comprehensively covering seven reasoning types: abductive, analogical, causal, deductive, inductive, spatial, and temporal. Unlike existing reasoning benchmarks, MMR-Life does not rely on domain-specific expertise but instead requires models to integrate information across multiple images and apply diverse reasoning abilities. The evaluation of 37 advanced models highlights the substantial challenge posed by MMR-Life. Even top models like GPT-5 achieve only 58% accuracy and display considerable variance in performance across reasoning types. Moreover, we analyze the reasoning paradigms of existing MLLMs, exploring how factors such as thinking length, reasoning method, and reasoning type affect their performance. In summary, MMR-Life establishes a comprehensive foundation for evaluating, analyzing, and improving the next generation of multimodal reasoning systems.

1 INTRODUCTION

MMR-Life addresses the gap between existing benchmarks and everyday multimodal reasoning by testing how models draw conclusions across images from real-life scenarios. Its evaluations reveal substantial performance limits and differences across reasoning types, alongside findings about MLLM reasoning paradigms.

Introduction

  • Prior benchmarks diverge from everyday reasoning through expert-knowledge or synthetic-puzzle tasks, single-image inputs, or multi-image tasks limited to non-reasoning or narrow reasoning types.The paper notes that symbolic puzzle images differ from real-world scenes and that everyday visual information is encountered as sequences of images.Source: §1 · p. 2
  • MMR-Life comprises 2,646 questions across seven reasoning types, requiring models to integrate multiple real-world images without domain-specific expertise.Questions ask models to extract key information from the images and derive new conclusions.Source: §1 · p. 2
  • GPT-5 reaches 58.69% accuracy and Gemini-2.5-Pro 56.86% on MMR-Life, both falling short of human performance by 14%.The evaluation covers 37 advanced MLLMs.Source: §1 · p. 2
  • Models perform relatively well on analogical, deductive, and inductive reasoning but face bottlenecks in causal, spatial, and temporal reasoning.The benchmark evaluation shows substantial performance disparities across reasoning types.Source: §1 · p. 2
  • The analysis finds that long thinking benefits only limited reasoning types, RL generalizes less effectively in small models, and reasoning types cluster into patterns.These findings are presented as insights into existing MLLM reasoning paradigms.Source: §1 · pp. 2–3

2 THE MMR-LIFE BENCHMARK

MMR-Life evaluates multimodal reasoning across everyday, real-life scenarios using multi-image questions spanning seven reasoning types and 21 tasks. Its curation pipeline combines varied image sources, task-specific question and option construction, and filtering to distinguish it from benchmarks based on symbolic images or reused questions.

2.1 OVERVIEW

  • MMR-Life comprises 2,646 multiple-choice questions based on 19,108 real-life images, covering seven reasoning types and 21 tasks.The tasks draw on multi-image sets from contexts such as domestic life, dining, and sports.Source: §2.1 · p. 3

2.2 DATA CURATION PIPELINE

  • Images come from public datasets, web resources, video frames, and existing multi-image or video reasoning benchmarks.The pipeline selects related images, uses publicly available real-world resources, and removes low-quality images; video frames are chosen for clarity.Source: §2.2 · pp. 3–4
  • The benchmark defines abductive, analogical, causal, deductive, inductive, spatial, and temporal reasoning to represent varied everyday situations.These types cover explanation, analogy, cause-to-effect, rule-based deduction, pattern generalization, spatial relations, and event order or timing.Source: §2.2 · p. 4
  • Question-answer pairs are synthesized automatically when image information suffices or manually annotated when tasks require reasoning about implicit information.This process produces a diverse set of 3.2K questions from multiple sources.Source: §2.2 · p. 4
  • Questions use five options, with incorrect image choices sampled heuristically and incorrect text choices generated by models and manually selected.For text options, GPT-5-mini (OpenAI, 2025b), GPT-4o (OpenAI, 2024), and Qwen2.5-VL-32B (Bai et al., 2025) generate responses; annotators select four high-quality incorrect options.Source: §2.2 · p. 4
  • Three filtering steps remove questions that are too easy, options with substantial format differences, and items with ambiguity, multiple correct answers, or domain-specific requirements.Questions answered correctly by all three smaller models are filtered; co-authors conduct the final quality review.Source: §2.2 · p. 4

2.3 COMPARISONS WITH EXISTING BENCHMARKS

  • MMR-Life excludes symbolic images and uses newly annotated questions rather than items sampled directly from existing datasets, textbooks, or the web.The paper presents these choices as aligning the benchmark with natural images encountered in daily life and reducing data-contamination risk.Source: §2.3 · p. 4

3 MAIN EXPERIMENT

MMR-Life remains challenging for current MLLMs: GPT-5 reaches 58.69% accuracy, performance varies substantially by reasoning type, and thinking mode offers limited gains for open-source models.

3.1 EXPERIMENTAL SETTINGS

  • The evaluation covers 37 models, including non-thinking and thinking MLLMs, with a 210-question human mini-test answered by 12 students.The human test yielded 600 valid answers, with students instructed not to use external knowledge sources.Source: §3.1 · p. 5
  • All models receive the same zero-shot CoT prompt, and open-source models are evaluated over five runs with average performance reported.Experiments use 8 NVIDIA A100 GPUs.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • 58.69% accuracy is GPT-5’s MMR-Life result, 14% below human performance, underscoring the benchmark’s challenge for MLLMs.Almost all open-source models score below 40%, and Skywork-R1V-38B and InternVL3.5-8B perform below random guessing (20%).Source: §3.2 · pp. 5–6, Table 3 · p. 6
  • Reasoning performance varies sharply: spatial accuracy peaks at 25.10% versus 79.76% for humans, while models perform well on analogy, deduction, and induction.Most closed-source models outperform humans on analogical reasoning; models struggle more with causal, spatial, and temporal tasks.Source: §3.2 · p. 6
  • Open-source thinking models average 27.15% accuracy, below the 29.01% average for open-source no-thinking models, while closed-source thinking models generally outperform their no-thinking counterparts.The results indicate that adding thinking mode does not yet improve open-source models’ reasoning on MMR-Life.Source: §3.2 · p. 6

4 THINKING PATTERN ANALYSIS

The analysis finds that longer reasoning does not reliably improve every reasoning type, enhancement methods generalize unevenly, and reasoning categories show both shared and distinct patterns. Together, these findings highlight substantial variation in how MLLMs reason across tasks.

4.1 IS LONGER THINKING ALWAYS BETTER?

  • Across 14 models, longer outputs tend to accompany higher accuracy roughly logarithmically, but CoT or larger reasoning budgets can hurt induction while helping analogy.Some open-source thinking models are exceptions to the overall trend, and the authors suggest longer CoT may suit step-by-step tasks better than tasks benefiting from faster thinking.Source: §4.1 · p. 7

4.2 DO GENERALIZABLE REASONING ENHANCEMENT METHODS EXIST?

  • Across 7B–72B models, enhancement methods’ average performance gap from CoT shrinks and more subtypes regress; on Qwen-2.5-VL-72B, BoN and GRPO underperform CoT.The authors hypothesize that these methods improve sampling efficiency toward correct paths, leaving less room for gains when larger models are already more likely to sample them.Source: §4.2 · p. 8, Table 4 · p. 8
  • GRPO generalizes worse than BoN on all three tested models, and additional small-model experiments find RL-trained models consistently below BoN@8 on base models.Some RL models also perform worse than their base models using CoT; the authors leave open whether RL on small models overfits to specific datasets.Source: §4.2 · pp. 8–9, Figure 6 · p. 8

4.3 DO DIFFERENT REASONING TYPES CORRELATE?

  • Reasoning-type correlations range from 0.97 for inductive–analogical reasoning to 0.40 for spatial–inductive reasoning, while clustering suggests higher-order patterns and spatial distinctiveness.The authors relate the analogical–inductive cluster to abstracting general rules from concrete features, and spatial reasoning’s distance to its location and distance-estimation demands.Source: Figure 7 · p. 8, §4.3 · p. 9

5 ERROR ANALYSIS

An analysis of incorrect responses from GPT-5 and Gemini-2.5-Pro finds reasoning errors most common, alongside notable abstraction, knowledge, and perception failures. The findings expose shortcomings across reasoning and image understanding that may inform future MLLMs.

5 ERROR ANALYSIS

  • Reasoning errors account for 32% of failures, including temporal confusion (42%), causal inversion (24%), and missing key steps (24%).The error distribution is shown in Figure 8.Source: §5 · p. 9
  • Abstraction errors account for 17% of failures, while knowledge errors account for 17% and perception errors for 12%.Abstraction errors reflect short-term thinking such as making associations; knowledge errors involve recalling knowledge, and perception errors include identifying static attributes and dynamic changes.Source: §5 · p. 9
  • The analysis randomly samples 20 incorrect examples per reasoning type for each of the two strongest MMR-Life models, GPT-5 and Gemini-2.5-Pro.Appendix H presents a selection of 42 notable cases with detailed analyses.Source: §5 · p. 9

6 RELATED WORK

Related work spans methods for enhancing multimodal reasoning and benchmarks for evaluating it. MMR-Life evaluates diverse methods and MLLMs while addressing gaps in real-life reasoning coverage.

Multimodal Reasoning Enhancement Methods

  • Multimodal reasoning methods extend CoT with perception, use reward models to select reasoning paths, and include thinking MLLMs; MMR-Life evaluates these approaches and models.Examples include IPVR (Chen et al., 2023), CCoT (Mitra et al., 2024), VisualSketchpad (Hu et al., 2024), and thinking MLLMs such as VL-Rethinker (Wang et al., 2025b).Source: §6 · pp. 9–10

Multimodal Reasoning Benchmarks

  • Knowledge-heavy STEM and symbolic-image benchmarks diverge from real-life reasoning, while real-life spatial benchmarks cover few types; MMR-Life uses real-life multi-image inputs across seven types.The cited benchmark groups include GPQA (Rein et al., 2023) and PuzzleVQA (Chia et al., 2024); prior spatial work includes Yang et al. (2025a) and Li et al. (2025b).Source: §6 · p. 10

7 CONCLUSION … C DETAILS OF ANNOTATION PROTOCOLS

MMR-Life evaluates MLLMs’ reasoning across seven types using multiple real-life images, and the authors report substantial challenges and performance imbalances while analyzing factors related to reasoning ability.

7 CONCLUSION

  • MMR-Life evaluates seven reasoning types across real-life scenarios with multiple images, and existing MLLMs still face significant challenges and performance imbalances.The authors also analyze relationships between thinking length, enhancement methods, and reasoning abilities.Source: §7 · p. 10

ETHICS STATEMENT

  • The authors state that data curation followed copyright and licensing restrictions, excluded private or harmful images, and used volunteer participants rather than paid annotation platforms.Participants understood the research goals, procedures, and intended data use; the authors state the benchmark does not replace original sources or affect their market value or utility.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Reproducibility materials include descriptions and links for data sources, 210 supplementary items, construction details, experimental settings and prompts, and code.The authors commit to open-sourcing all data and code if the paper is accepted.Source: REPRODUCIBILITY STATEMENT · p. 10

A THE USE OF LARGE LANGUAGE MODELS

B KEY CONCEPTS IN MMR-LIFE

B.1 REASONING IN REAL-LIFE SCENARIO

  • Real-life scenario reasoning uses multiple natural or realistically simulated images and textual descriptions, and should be solvable with commonsense rather than specialized knowledge.The benchmark excludes purely abstract diagrams and symbolic renderings; this definition guides this study’s design rather than claiming broad applicability.Source: §B.1 · p. 20

B.2 MULTI-IMAGE VS. VIDEO

  • The authors chose multiple images over video because videos limit reasoning types and data diversity, while sampled frames can add irrelevant noise.Video relationships typically emphasize temporal sequencing, making parallel-image tasks such as analogy or induction difficult to design.Source: §B.2 · pp. 20–21

C DETAILS OF ANNOTATION PROTOCOLS

C.1 DATA SOURCES OF DIFFERENT TASKS … D DATA DIVERSITY OF MMR-LIFE

The appendix documents MMR-Life’s task-wise data sources and annotation constraints, as well as negative-option generation prompts and visualizations of dataset diversity.

C.1 DATA SOURCES OF DIFFERENT TASKS

  • Table 5 presents the data sources for every MMR-Life task.The collection process required annotators to follow copyright and licensing rules on source sites and datasets.Source: §C.1 · p. 21
  • The dataset includes only natural images, excluding symbolic diagrams and other non-photographic forms.This restriction follows Definition 1.Source: §C.1 · p. 21

C.2 ANNOTATION GUIDELINES

  • Questions had to use at least two images, be written in English, and be solvable without complex domain-specific knowledge.These were among the guidelines given to annotators writing questions and golden answers.Source: §C.2 · p. 21
  • Questions had to be unambiguous, answerable with one option, and clearly conform to their reasoning-type definitions.The guidelines aimed to distinguish tasks across reasoning types.Source: §C.2 · p. 21

C.3 PROMPTS FOR NEGATIVE OPTION GENERATION

  • The paper lists negative-option generation prompts in Figures 11–17.The passage identifies the figures containing the prompts but does not describe their contents.Source: §C.3 · p. 21

D DATA DIVERSITY OF MMR-LIFE

  • Figure 9 visualizes image-type variety, while Figure 10 presents the distribution of input image counts.Appendix E illustrates the different task types included in the study.Source: D DATA DIVERSITY OF MMR-LIFE · p. 22

E TASK DETAILS … F DETAILS OF MAIN EXPERIMENT

The task descriptions operationalize seven kinds of reasoning through varied image-based problems, from inferring causes and patterns to planning movements and reconstructing sequences. Each task asks models to use visual context, relationships, or changes over time to reach a specified conclusion.

E.1 ABDUCTIVE REASONING

  • Abductive tasks infer motivations or triggering causes from behavior, interaction, and physical collision clues.E.1.1 HUMAN ACTIVITY ATTRIBUTION infers motivation from environmental and behavioral cues; E.1.2 CHARACTER INTERACTION ATTRIBUTION infers causes of events from character behavior and situation; E.1.3 MULTI-HOP COLLISION ATTRIBUTION traces collision chains to identify a root cause.Source: §E.1.1 · p. 22, §E.1.2 · p. 22, §E.1.3 · p. 22

E.2 ANALOGICAL REASONING

  • Analogical tasks transfer visual relationships or learned preferences and styles to select matching animals, products, or artwork.E.2.1 ANIMAL RELATION INFERENCE matches a relation between animal pairs; E.2.2 PRODUCT SIMILARITY INFERENCE uses owned or disliked products to infer preferences; E.2.3 ARTWORK STYLE INFERENCE identifies work matching an artist’s learned style.Source: §E.2.1 · p. 22, §E.2.2 · p. 22, §E.2.3 · p. 23

E.3 CAUSAL REASONING

  • Causal tasks predict character reactions, collision propagation, or altered fluid flow under a changed condition.E.3.1 CHARACTER INTERACTION PREDICTION uses context and character relations to predict a follow-up; E.3.2 MULTI-HOP COLLISION PREDICTION simulates collision chains from image sequences; E.3.3 COUNTERFACTUAL FLUID PREDICTION predicts flow paths and final positions after barrier removal.Source: §E.3.1 · p. 23, §E.3.2 · p. 23, §E.3.3 · p. 23

E.4 DEDUCTIVE REASONING

  • Deductive tasks apply constraints, rules, or process logic to determine production counts, poker winners, and cooking order.E.4.1 MATERIAL COMPOSITION DEDUCTION calculates producible products from component quantities and requirements; E.4.2 CARD WINNER DEDUCTION applies Texas Hold ’em rules to determine the best hand and winner; E.4.3 RECIPE STEP DEDUCTION orders preparation images using ingredient states, tools, and causal relationships.Source: §E.4.1 · p. 23, §E.4.2 · p. 23, §E.4.3 · p. 23

E.5 INDUCTIVE REASONING

  • Inductive tasks extrapolate patterns from bird distributions, diseased leaves, or sports characteristics.E.5.1 BIRD MIGRATION INDUCTION infers migration patterns and predicts an upcoming-year distribution; E.5.2 PLANT DISEASE INDUCTION learns visual disease features from leaf samples; E.5.3 SPORT FEATURE INDUCTION identifies the next sport matching observed characteristic relationships.Source: §E.5.1 · p. 24, §E.5.2 · p. 24, §E.5.3 · p. 24

E.6 SPATIAL REASONING

  • Spatial tasks infer indoor object directions, estimate camera rotations, or plan an obstacle-avoiding route.E.6.1 RELATIVE POSITION ESTIMATION derives object positions and directions; E.6.2 CAMERA ROTATION ESTIMATION estimates rotation angles and directions across viewpoints; E.6.3 NAVIGATION ROUTE PLANNING selects moves using only 90° or 180° turns and forward motion while avoiding obstacles.Source: §E.6.1 · p. 24, §E.6.2 · p. 24, §E.6.3 · p. 24

E.7 TEMPORAL REASONING

  • Temporal tasks order crowd images, predict a driving scene’s next frame, or locate an activity within a video.E.7.1 CROWD TIMELINE RECONSTRUCTION uses positions, actions, and environmental changes to order images; E.7.2 DRIVING SEQUENCE PREDICTION integrates road, vehicle, traffic, and environmental cues; E.7.3 HUMAN ACTIVITY LOCALIZATION assigns an activity to the start, middle, end, or entirety of a video.Source: §E.7.1 · p. 25, §E.7.2 · p. 25, §E.7.3 · p. 25

F.1 DETAILED EXPERIMENTAL SETUP · F.2 FULL EXPERIMENTAL RESULTS · F.3 EXPERIMENTAL RESULTS ON TINY SET

The appendix specifies the MLLM evaluation setup and points readers to full results for 37 models and a separate mini-test-set comparison. The supplied passages identify the results tables but do not report their scores.

F.1 DETAILED EXPERIMENTAL SETUP

  • The evaluation roster includes dated OpenAI and Anthropic model versions, including GPT-5, GPT-4.1, GPT-4o, and Claude Sonnet variants.Source: §F.1 · p. 25
  • The experiment’s multimodal-model roster includes Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct, Qwen2.5-VL-72B-Instruct, VL-Rethinker-7B, VL-Rethinker-32B, VL-Rethinker-72B, InternVL3.5-8B, InternVL3.5-30B-A3B, InternVL3.5-38B, gemma-3-4b-it, gemma-3-12b-it, and gemma-3-27b-it.Source: §F.1 · p. 25
  • Inference used temperature 0.5, top p 0.5, and seed 17.Source: §F.1 · p. 25
  • The main-experiment prompt is shown in Figure 18.Source: §F.1 · p. 25

F.2 FULL EXPERIMENTAL RESULTS

  • Table 6 presents full evaluation results for 37 MLLMs.Source: §F.2 · p. 25

F.3 EXPERIMENTAL RESULTS ON TINY SET

  • Table 7 presents a model-performance comparison on the mini test set.Source: §F.3 · p. 26

G DETAILS OF THINKING PATTERN ANALYSIS

The thinking-pattern analysis uses base and RL-version models trained on various datasets to illustrate the generalizability of its conclusions.

G DETAILS OF THINKING PATTERN ANALYSIS

  • The analysis uses four CoT-prompted base models and four models corresponding to their RL-training versions to illustrate conclusion generalizability.The base models are MiMo-VL-7B-SFT, RL-Onevision, Qwen-2.5-VL-32B, and Qwen-2.5-VL-7B; the RL versions are MiMo-VL-7B-RL, RL-Onevision-RL, MM-Eureka-32B, and VL-Rethinker-7B.Source: G DETAILS OF THINKING PATTERN ANALYSIS · p. 26

H CASE STUDY … Temporal：Driving Sequence Prediction

This case-study section supplements benchmark results with task-level examples across several reasoning categories, including correct and incorrect responses by GPT-5 and Gemini-2.5-Pro. It illustrates temporal reasoning through driving-sequence prediction and human-activity localization.

H CASE STUDY

  • Abductive case studies illustrate human-activity and multi-hop collision attribution, with both correct and incorrect responses by GPT-5 and Gemini-2.5-Pro.Source: H CASE STUDY · pp. 26, 37, 41

Analogical：Product Similarity Inference

Analogical：Artwork Style Inference

Deductive：Material Composition Deduction

Deductive：Card Winner Deduction

Inductive：Sport Feature Induction

Spatial：Camera Rotation Estimation

Temporal：Driving Sequence Prediction

Related papers

Loading 2603.02024v1…