Source-linked AI summary
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, Yu Cheng
TL;DR
MLLMs lack well-suited benchmarks for evaluating integrated reasoning over text and images. The paper introduces EMMA, a cross-domain benchmark designed around genuinely multimodal, multi-step problems, and finds substantial performance gaps that prompting and compute scaling only marginally reduce.
Problem
Existing multimodal benchmarks often emphasize surface visual understanding, textual recall, or questions solvable through language alone, leaving integrated multimodal reasoning under-evaluated.
Method
EMMA filters existing problems using text and image captions, then adds and labels multimodal reasoning questions across math, physics, chemistry, and coding.
Results
All evaluated MLLMs perform suboptimally on EMMA; o1 reaches 45.75% on the balanced subset, while CoT and test-time scaling provide only limited gains against human performance.
Takeaways & Limitations
EMMA indicates that current MLLMs need architectures and training paradigms better able to integrate and reason over diverse modalities.
Takeaways & Limitations
The benchmark’s physics section is underrepresented and its chemistry section could be broadened in future iterations.
Abstract
from arXiv · showhide
The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
1. Introduction
Multimodal reasoning requires coordinated use of visual and linguistic information, but existing benchmarks often fail to test this integration. EMMA targets cross-modal, multi-step reasoning and exposes substantial limitations in current MLLMs.
- Text and visual reasoning jointly support technical problem-solving because some mathematical ideas cannot be fully addressed by either modality alone.
- Existing benchmarks often test surface-level visual understanding or textual knowledge, while many image-based questions remain solvable through language alone.
- EMMA introduces problems requiring back-and-forth interaction between visual interpretation and multimodal reasoning rather than independent reasoning in each modality.
- EMMA spans domain-specific challenges including 3D transformations, chemical structure recognition, physical simulations, and program-output visualization.The benchmark combines 992 filtered questions from existing benchmarks with 1,796 newly constructed questions.
- 45.75% is o1’s accuracy on EMMA’s balanced subset, 8.5% above Qwen2-VL and 32% below human experts.All evaluated models perform suboptimally regardless of Chain-of-Thought prompting.
- Textual Chain-of-Thought and test-time compute scaling do little to resolve models’ inability to generate valid visual reasoning steps.The reported limitation is especially pronounced for fine-grained spatial understanding and multi-step reasoning.
2. Related Work
Prior work has advanced multimodal modeling and reasoning benchmarks, but many benchmarks still contain redundant text and image information. EMMA instead emphasizes tasks requiring strong visual reasoning.
- Modern MLLMs build on advances in vision-language modeling, LLMs, visual instruction tuning, and increasingly available training data.
- Recent reasoning research has expanded from text-only LLM tasks toward multimodal approaches such as Visual Chain-of-Thought.
- Multimodal benchmarks now cover spatial, relational, and domain-knowledge reasoning, but many remain reducible to text because images duplicate textual information.
- EMMA emphasizes multimodal reasoning challenges that are difficult to solve through text-based reasoning or a single visual pass.
3. The EMMA Benchmark
EMMA is a 2,788-question benchmark spanning four domains and fine-grained multimodal skills. Its curation filters out text- and caption-solvable problems, then expands the remaining taxonomies with manually constructed questions.
- The EMMA Benchmark: EMMA contains 2,788 problems across math, physics, chemistry, and coding, including 1,796 newly constructed questions.
- The EMMA Benchmark: Each problem receives fine-grained skill labels created or assigned by domain experts and verified through expert review.
- Data Curation: The curation pipeline removes existing questions solvable from text and image captions, then categorizes survivors and adds aligned samples.
- Data Curation: Math contributes 892 questions organized into categories such as 3D Simulation, 2D Transformation, Path Tracing, counting, and pattern inference.
- Data Curation: Physics retains 80 filtered problems and adds 76 manually collected problems spanning 3D field simulation, graph reasoning, and path tracing.
- Data Curation: Chemistry adds questions on chemical structures, bond counting, structure simulation, and reaction simulation using molecular data, RDKit, and chemistry experts.
- Data Curation: Coding questions are manually constructed to evaluate visualization creation and reproduction, including multiple-choice selection of chart-generating code.
4. Experiments
Experiments evaluate nine MLLMs on EMMA and EMMA-mini, comparing prompting and test-time scaling strategies. Models remain substantially below human performance, although scaling and stronger reward models provide limited gains.
- Main Results: On EMMA-mini, o1 achieves 45.75% accuracy, trailing human experts by 32%.
- Main Results: Across the full benchmark, closed-source models generally outperform open-source models, particularly with Chain-of-Thought prompting.
- Prompting Strategies: Chain-of-Thought prompting generally improves closed-source models but reduces open-source performance, with leading open-source models losing over 6% on EMMA and EMMA-mini.
- Test-Time Compute Scaling: Test-time scaling improves accuracy but fails to close the human-performance gap; the largest gains are 5.25% for GPT-4o, 4.5% for Gemini 2.0 Flash, and 7.5% for Gemini 2.0 Flash Thinking.
- Test-Time Compute Scaling: Gemini 2.0 Flash Thinking overtakes o1 by 5% with majority voting at N=16, yet its best performance still trails humans by 27%.
- Test-Time Compute Scaling: Stronger reward models and tournament-style selection improve scaling, while self-reward modeling remains below majority voting.
5. Error Analysis
Error analysis indicates that visual reasoning is the dominant failure mode for o1, while textual Chain-of-Thought helps unevenly across task types. Visual-centric tasks appear poorly suited to textual reasoning traces.
- Error Distribution: Visual reasoning errors constitute 52.83% of o1’s math and coding errors, the largest error category.
- Error Distribution: Perceptual errors account for 30.19% of errors, while textual reasoning and knowledge errors account for 9.43% and 7.55%, respectively.
- CoT Effects: CoT prompting improves reasoning in closed-source MLLMs but tends to reduce performance in open-source models.
- CoT Effects: Textual CoT is conjectured to help Multi-Hop Object Counting more than 2D Transformation because relative positions can be described linguistically.
- CoT Effects: GPT-4o answers a spatial problem correctly without CoT but incorrectly with CoT, showing superficial associations rather than genuine visual reasoning.
6. Conclusion
EMMA benchmarks advanced cross-modal reasoning across multiple technical domains and exposes a substantial gap between current MLLMs and human experts. The authors conclude that improved multimodal architectures and training paradigms are needed, while identifying areas for benchmark expansion.
- Conclusion: EMMA contains multimodal questions requiring cross-modal reasoning that cannot be solved by reasoning independently within each modality.
- Conclusion: Evaluation of nine state-of-the-art MLLMs reveals a substantial performance gap from human experts, with CoT and test-time scaling offering only marginal gains.
- Conclusion: EMMA highlights the need for architectures and training paradigms that better integrate and reason over diverse modalities.
- Conclusion: Future benchmark iterations could enrich the underrepresented physics section and broaden chemistry coverage.
B.1. Composition of EMMA
EMMA spans mathematics, physics, chemistry, and coding through categories designed to test visual, textual, and iterative multimodal reasoning. Its questions include spatial transformations, physical simulations, chemical structure and reaction reasoning, and visualization-program reasoning.
- EMMA covers math, physics, chemistry, and coding with fine-grained categories for multimodal reasoning skills.The benchmark provides subject-specific breakdowns and labels for the abilities assessed by each question.
- Math: Math includes 2D and 3D transformations, path tracing, multi-hop visual counting, and pattern inference.The math portion contains 892 questions distributed across five categories.
- Physics: Physics tasks require visual decomposition, three-dimensional field simulation, graph reasoning, multi-hop reasoning, and path tracing.The 156 physics questions include simulations of forces, fields, graphs, circuits, and refracted particle trajectories.
- Chemistry: Chemistry ranges from bond counting and molecular structure recognition to reaction simulation and graph reasoning.Reaction Simulation requires inferring post-reaction molecular expressions from structures and electron-flow arrows; Reaction Simulation Pro compares image-based answer choices.
- Coding: Coding evaluates visualization reasoning through selecting code from images, selecting images from code, and modifying visualization programs.The four tasks contain 188, 188, 94, and 94 questions respectively, with questions often assigned multiple skill categories.
B.2. Additional Data Curation Details
The curation process combines manually designed visualization questions with filtering and verification procedures for chemistry and coding. Coding questions are built from seed visualizations, variations, and task-specific comparisons, while chemistry emphasizes increasingly difficult molecular reasoning.
- Chemistry: Chemistry filters out questions answerable from captions and constructs a new three-task suite from scratch.The suite progresses from knowledge-based counting to structure recognition and reaction simulation.
- Chemistry: Chemistry reaction-simulation answers are constructed and verified by a PhD candidate in chemical molecules.The reaction questions are sourced from a collection of chemical reactions.
- Coding: Coding curation begins by identifying seed visualizations using advanced techniques or rich design variation.Sources include CharXiv, the official matplotlib gallery, and the authors’ prior experience.
- Coding: Each coding seed produces four variations, including flawed MLLM-generated visualizations that models must distinguish from correct outputs.The process targets recognition of ill-formed or nonsensical visualizations as well as reconstruction ability.
- Coding: The resulting coding set contains 564 questions evenly divided among visualization selection and modification tasks.Questions are constructed from 188 visualizations across the curated sets.
B.3. Comparison with Other Benchmarks
EMMA is designed to emphasize multimodal reasoning that depends on visual information rather than surface cues or text-only solutions. It adds fine-grained taxonomies and new visualization tasks while expanding underrepresented science problems.
- Math: Existing math benchmarks often contain shallow visual cues or questions solvable through text-dominant reasoning.Some image-based math tasks can be answered from captions alone.
- Math: EMMA adds a manually verified math taxonomy focused on graphic transformation and spatial simulation.The categories are reported as applicable to EMMA and adaptable to datasets such as MMMU.
- Science: Only 100 science problems remained after filtering relevant benchmarks, and newly constructed questions expanded the set to 1,332.The science curation addresses the limited multimodal physics and chemistry coverage of prior benchmarks.
- EMMA filters and reviews questions to ensure that visual information is important for solving them.The benchmark targets problems where visual imagination, simulation, molecule reasoning, or reaction reasoning is required.
- Coding: EMMA extends visualization evaluation with Vis2Code, Code2Vis, and visualization-modification tasks.Its coding questions are multiple-choice and include fine-grained expert-generated skill categories.
C.4. Breakdown of Experiment Results by Category
The experiment results are broken down by question category across the four EMMA subjects. Separate tables report the category-level results for math, physics, chemistry, and coding.
- Results are reported separately for math, physics, chemistry, and coding.
- Each subject’s results are presented as a breakdown across its categories.
- Tables 7 through 10 contain the category-level results for the four subjects.Table 7 covers math, Table 8 physics, Table 9 chemistry, and Table 10 coding.
C.5. Best-of-N With a Specialized Math Reward Model
This section compares a specialized math reward model with generalist reward models for best-of-N evaluation on EMMA-mini. Generalist reward models provide better rewards overall, while the specialized model’s non-multimodal design limits its suitability for this benchmark.
- Evaluation setup: GPT-4o and Gemini 2.0 Flash responses are scored with Qwen2.5-Math-RM-72B, selecting the highest-scoring answer across N values of 2, 4, 8, and 16.Because the reward model does not accept images, GPT-4o-generated captions are supplied for every image.
- Results: Generalist reward models generate better rewards than Qwen2.5-Math-RM-72B on the math portion of EMMA-mini.The comparison is reported in Table 11 as an overall result across the evaluated reward models.
- Caveat: Qwen2.5-Math-RM-72B is not a text-only model, which likely affects its performance on a multimodal benchmark like EMMA.This caveat qualifies the comparison between the specialized math reward model and generalist reward models.
Error Analysis:
The case studies show that MLLM errors arise when tasks require detailed visual simulation, structural interpretation, or careful coordination between code and images. Examples span chemistry, physics, math, and coding, including both incorrect reasoning and successful decomposition.
- Chemistry: In chemistry, reaction simulation requires inferring the post-reaction SMILES expression from molecular structure and electron-flow arrows.The example asks readers to select the molecular image corresponding to the molecule after electron relocation.
- Chemistry: o1 explains a 17-bond chemistry answer by decomposing the structure into a cyclohexyl moiety, an acyl-center region, and an aromatic component.The response counts bonds by structural parts while excluding bonds involving hydrogen.
- Coding: GPT-4o incorrectly assumes legend markers inherit the scatter plot color because repeated plt.scatter() calls assign distinct marker colors.The error concerns the interaction between code execution behavior and the rendered visualization.
- Coding: GPT-4o hallucinates multiple code–image disagreements while missing the actual discrepancy: default padding at both ends of the intensity line chart.The case illustrates over-complication during multimodal code and visualization comparison.