PaperlayerView exact revision on arXiv

Source-linked AI summary

ReMI: A Dataset for Reasoning with Multiple Images

Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo, Sreenivas Gollapudi, Ahmed Qureshi

arXiv:2406.09175v1cs.CVcs.CL

TL;DR

Existing vision-language benchmarks predominantly use single images and do not directly measure complex reasoning across multiple images. ReMI is a 13-task benchmark covering varied domains and task properties; models remain substantially behind humans on it.

  • Problem

    Existing vision-language benchmarks predominantly use single images and do not directly measure how well models integrate information across images.

    Source: Related Work · p. 3

  • Method

    ReMI is a 13-task benchmark covering multiple reasoning domains and properties of multi-image problems, with tasks requiring at least two images.

    Source: Introduction · p. 2

  • Results

    Models remain substantially behind human performance on ReMI, although separate-image inputs often improve performance, especially when interleaved with question text.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The results show substantial room to improve current state-of-the-art LLM reasoning capabilities on multi-image tasks.

    Source: Conclusion · p. 11

    ReMI spans many domains but does not cover all settings where multi-image reasoning may be needed, including chemistry and music sheets.

    Source: Limitations · p. 22

Abstract

from arXiv · show

With the continuous advancement of large language models (LLMs), it is essential to create new benchmarks to effectively evaluate their expanding capabilities and identify areas for improvement. This work focuses on multi-image reasoning, an emerging capability in state-of-the-art LLMs. We introduce ReMI, a dataset designed to assess LLMs' ability to Reason with Multiple Images. This dataset encompasses a diverse range of tasks, spanning various reasoning domains such as math, physics, logic, code, table/chart understanding, and spatial and temporal reasoning. It also covers a broad spectrum of characteristics found in multi-image reasoning scenarios. We have benchmarked several cutting-edge LLMs using ReMI and found a substantial gap between their performance and human-level proficiency. This highlights the challenges in multi-image reasoning and the need for further research. Our analysis also reveals the strengths and weaknesses of different models, shedding light on the types of reasoning that are currently attainable and areas where future models require improvement. To foster further research in this area, we are releasing ReMI publicly: https://huggingface.co/datasets/mehrankazemi/ReMI.

Introduction

Multi-image reasoning requires combining information across text and multiple images, but existing benchmarks do not extensively evaluate it. ReMI covers diverse reasoning tasks and properties, and current models remain substantially behind humans.

Introduction

  • Multi-image reasoning requires step-by-step solutions that combine text with information from multiple images, a skill not extensively evaluated by existing benchmarks.Many current multimodal benchmarks are based on a single image.Source: Introduction · pp. 1–2
  • ReMI’s 13 tasks span diverse reasoning domains and properties, require at least two and up to six images, and use heterogeneous image types.The tasks cover domains including algebra, geometry, physics, charts, coding, logic, and spatial and temporal reasoning.Source: Introduction · p. 2
  • Models remain substantially behind human performance on ReMI, and separate-image inputs often improve performance, especially when images are interleaved with question text.The authors also report a failure analysis intended to guide future improvement efforts.Source: Introduction · p. 2

Related Work

Prior vision-language benchmarks predominantly test single-image inputs, while existing cross-image benchmarks are restricted in task type or domain. ReMI targets complex reasoning across multiple images.

Related Work

  • Frontier vision-language models can condition on sequences of interleaved image and text, but most vision-language benchmarks still test a single image-text pair.Earlier vision-language models added visual encoders and attention layers to language models.Source: Related Work · p. 2
  • Reasoning benchmarks cover science, mathematical problem solving, and logical deduction, while ReMI focuses on complex reasoning across multiple images.The related-work discussion also identifies broad reasoning suites containing multiple reasoning tasks.Source: Related Work · pp. 2–3
  • Existing vision-language benchmarks predominantly use single images and do not directly measure integration of information across images.Some cross-image benchmarks exist, but they are limited to entailment or a narrower set of domains.Source: Related Work · p. 3

The ReMI Dataset

ReMI comprises 13 tasks spanning reasoning domains and properties that arise when models work across multiple images. Its tasks require parsing image and question information and reasoning toward an answer.

The ReMI Dataset

  • ReMI includes 13 tasks across domains such as algebra, calculus, geometry, physics, temporal and spatial reasoning, charts, maps, graph theory, and coding.The benchmark aims to cover key multi-image properties as well as a wide range of domains.Source: The ReMI Dataset · p. 3
  • Tasks vary in whether images must be consumed sequentially or as a set, show the same or different concepts, and are interleaved with text or presented first.Some tasks also vary the number of input images.Source: The ReMI Dataset · pp. 3, 5
  • Solving ReMI tasks requires parsing information in the images and question text, then reasoning with that information to reach an answer.Figure 2 illustrates task samples, while Table 1 specifies each task’s domain and properties.Source: The ReMI Dataset · p. 5
  • Sample tasks include reading multiple function graphs, relating geometric shapes, combining a shape with a cost table, and reasoning about collisions, clocks, schedules, and charts.The chart task can require comparing values across different chart formats or reasoning jointly from both charts.Source: The ReMI Dataset · p. 5
  • Other tasks ask models to edit TikZ code using rendered images, compare graphs for isomorphism, match map routes to a description, identify a target in an image, or complete a shape matrix.These tasks use multiple images or image-and-text inputs to pose their reasoning problems.Source: The ReMI Dataset · p. 5

Experiments

ReMI evaluations show that current models remain well behind human performance, with performance varying across tasks and models. The experiments also find gains from providing images separately, identify diverse failure sources, and measure improvements from few-shot examples.

Experiments

  • Experiments report accuracy, use a 1% tolerance for numeric answers, and sample 20 test examples per task for human performance.The naive baseline guesses from question text; the reported models include Gemini and GPT-4 families and Claude 3 Sonnet.Source: Experiments · pp. 5–6

Human Baseline Substantially Beats SoTA Models in Multi-Image Reasoning

  • Models substantially outperform the naive baseline but generally trail humans; Clocks and Isomorphism remain difficult, and model strengths vary by task.Gemini 1.5 substantially outperforms GPT4-Turbo on IQ, while GPT4-Turbo substantially outperforms Gemini 1.5 on EmojiAlgebra.Source: Table 2 · p. 6, Human Baseline Substantially Beats… · p. 7

Single-Image vs Multi-Image Reasoning

  • Separate images yield positive ERP on most tasks, averaging 19.8% gain for interleaved tasks versus 4.9% for non-interleaved tasks.Five of the six tasks most positively affected are interleaved; the figure compares multiple-image and single-image ERP for Gemini 1.5.Source: Single-Image vs Multi-Image Reasoning · p. 7, Figure 4 · p. 7

Failure Analysis

  • Manual review found task-specific failures including spatial selection errors, inconsistent IQ answers, chart misalignment, and geometry value misassignment.The analysis examined 20 incorrect examples per task; the reported failures also include code-edit errors despite some understanding of code structure.Source: Failure Analysis · pp. 7–10
  • Among 125 failed examples, 37.6% involved sound image reading but incorrect reasoning, 24.8% misreading with sound reasoning, 25.6% both, and 12% minor calculation errors.The reasoning-only category was the most frequent; Figure 6 presents the reasoning-error versus image-reading-error breakdown.Source: Figure 6 · p. 8, Failure Analysis · p. 10

Performance as a Function of Task Properties

  • Performance varies with task properties: frontier models do better on interleaved tasks, while sequence-versus-set and concept differences distinguish model performance.Gemini 1.5 does better on sequence tasks, whereas Claude 3 Sonnet does better on set tasks; GPT4 Turbo performs better when concepts are the same.Source: Figure 7 · p. 9, Performance as a Function of Task… · pp. 10–11

Zeroshot vs Fewshot Performance

Conclusion

ReMI is a dedicated benchmark for multi-image reasoning across domains and task properties. Frontier LLMs show a stark performance gap from humans, leaving substantial room for improvement.

Conclusion

  • ReMI is a dedicated benchmark covering multiple domains and key properties of multi-image reasoning.The conclusion frames it as an evaluation resource for reasoning with multiple images.Source: Conclusion · p. 11
  • Frontier LLMs show a stark gap from human performance on ReMI.The authors describe this as evidence of substantial room to improve current state-of-the-art reasoning capabilities.Source: Conclusion · p. 11
  • Future work can target limitations identified by failure analysis and measure whether improvements transfer to ReMI.Source: Conclusion · p. 11

Further Failure Analysis

The section supplements the main text’s high-level failure summaries with examples from several ReMI tasks.

Further Failure Analysis

  • Figures 9–17 provide examples of model failures across several tasks, complementing the main text’s high-level task summaries.Source: Further Failure Analysis · p. 15

Statistics about ReMI

The appendix reports statistics about question lengths, answer-label counts, and the number of images per ReMI problem.

Statistics about ReMI

Details about the Tasks in ReMI

The appendix describes task construction across mathematical, visual, temporal, spatial, coding, and logic problems, including synthetic generation, curated examples, and quality checks.

Details about the Tasks in ReMI

  • EmojiAlgebra uses image-based equations and expressions, while FuncRead asks questions about multiple plotted function graphs.FuncRead includes reading and combining values, limits, composition, derivatives, and matching graphs to functions.Source: Details about the Tasks in ReMI · p. 15
  • GeomShapes links two shape-formula computations through a shared hidden intermediate, while GeomCost combines shape formulas, story templates, and cost tables.GeomCost assigns cost values randomly from 1–100.Source: Details about the Tasks in ReMI · p. 15
  • Collisions varies two-object positions, masses, and velocities; clocks use generated time pairs, and schedules pair a clock with a varied train table.Collision examples assess elasticity and conservation; clock labels are time differences in minutes, and schedule questions ask for the next event.Source: Details about the Tasks in ReMI · pp. 15–17
  • Failure examples include incorrect clock/table readings, chart values, shape interpretation, and graph mappings, alongside cases where a correct answer follows faulty reasoning.Other examples show category-change, map-path, pattern, and code-editing errors.Source: Figure 10 · p. 16, Figure 11 · p. 16, Figure 12 · p. 17
  • Chart pairs come from edited data series or matrices rendered in varied chart styles, while CodeEdit removes a line of TikZ code and retains changed compilable images.Chart questions compare the two charts or detect differences; CodeEdit filters out unsuitable removed lines.Source: Details about the Tasks in ReMI · pp. 17–18
  • Isomorphism examples compare differently rendered graphs or graphs altered by adding or removing nodes or edges.The examples include positive isomorphic pairs and both easy and hard negative pairs.Source: Details about the Tasks in ReMI · pp. 18–19
  • Maps combines synthetic grid examples with real-world Google Maps examples, including counting and direction-matching queries.Synthetic directions are split across partial-path images; real examples are manually examined for distinguishing features.Source: Details about the Tasks in ReMI · pp. 19–20
  • RefCOCO examples use sampled points checked against referred-object boxes, while IQ problems ask models to apply a visual operation to complete a 2×2 grid.The authors also describe repeated author checks and a manual review of human-label disagreements.Source: Details about the Tasks in ReMI · pp. 20–21

Model Performance vs Human Time

The analysis relates model performance to the time humans spent per problem. Only Gemini Ultra and Gemini Flash show declining performance as human time increases; performance for the other models is nearly flat.

Model Performance vs Human Time

Experimental Setup

Models were evaluated with a 512-token output limit and prompted to return JSON containing step-by-step explanations and final answers. Final labels were compared after applying specified normalization and task-specific equivalences.

Experimental Setup

  • A 512-token output limit was used because longer responses often pursued a wrong path without recovering.Source: Experimental Setup · p. 21
  • Claude3 Sonnet, Gemini Ultra, Gemini Flash, Gemini 1.5, and GPT4 Turbo had invalid or premature responses at 0.4%, 0.3%, 0.5%, 0.8%, and 1.9%, respectively.Models were prompted for JSON fields containing a step-by-step explanation and final answer.Source: Experimental Setup · p. 21
  • Answer matching allowed specified formatting differences, including case, parentheses, units, spacing, code comments, and a function label versus its application.For FuncRead, a predicted f(x) counted as correct when the golden label was f.Source: Experimental Setup · p. 21

Limitations

The authors note that ReMI may omit domains requiring multi-image reasoning and that proxy-based analyses of task properties may shift slightly under a different procedure.

Limitations

  • ReMI covers many multi-image reasoning domains but may omit others, including chemical and music-sheet reasoning.Source: Limitations · p. 22
  • Task-property analyses use proxies to separate random chance from task difficulty.Source: Limitations · p. 22
  • A different procedure for separating chance and difficulty could change numeric results slightly, so broad patterns matter more than small differences.Source: Limitations · p. 22

Related papers

Loading 2406.09175v1…