PaperlayerView exact revision on arXiv

Source-linked AI summary

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, Honggang Zhang

arXiv:2407.01284v1cs.AIcs.CLcs.CVcs.LGcs.SC

TL;DR

Prior visual-math benchmarks largely assessed final answers, offering limited evaluation of the knowledge principles underlying LMM reasoning. WE-MATH decomposes problems by knowledge concepts and finds that most LMMs perform worse on multi-step than one-step problems.

  • Problem

    Previous visual-math benchmarks relied on end-to-end results, limiting assessment of inherent issues in LMMs’ reasoning processes.

    Source: §1 · p. 2

  • Method

    WE-MATH covers 6.5K problems across five granularity layers and 67 concepts, decomposing composite problems into sub-problems and evaluating reasoning through IK, IG, CM, and RM.

    Source: §1 · pp. 2–3

  • Results

    Most LMMs performed significantly worse on multi-step than one-step problems; GPT-4o accuracy fell from 72.84% to 43.64%.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    The authors identify insufficient knowledge as LMMs’ greatest vulnerability, while GPT-4o is the first LMM advancing toward the knowledge generalization stage.

    Source: §1 · p. 3, §5 · p. 12

Abstract

from arXiv · show

Visual mathematical reasoning, as a fundamental visual reasoning ability, has received widespread attention from the Large Multimodal Models (LMMs) community. Existing benchmarks, such as MathVista and MathVerse, focus more on the result-oriented performance but neglect the underlying principles in knowledge acquisition and generalization. Inspired by human-like mathematical reasoning, we introduce WE-MATH, the first benchmark specifically designed to explore the problem-solving principles beyond end-to-end performance. We meticulously collect and categorize 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and five layers of knowledge granularity. We decompose composite problems into sub-problems according to the required knowledge concepts and introduce a novel four-dimensional metric, namely Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of existing LMMs in visual mathematical reasoning and reveal a negative correlation between solving steps and problem-specific performance. We confirm the IK issue of LMMs can be effectively improved via knowledge augmentation strategies. More notably, the primary challenge of GPT-4o has significantly transitioned from IK to IG, establishing it as the first LMM advancing towards the knowledge generalization stage. In contrast, other LMMs exhibit a marked inclination towards Rote Memorization - they correctly solve composite problems involving multiple knowledge concepts yet fail to answer sub-problems. We anticipate that WE-MATH will open new pathways for advancements in visual mathematical reasoning for LMMs. The WE-MATH data and evaluation code are available at https://github.com/We-Math/We-Math.

1 Introduction

WE-MATH evaluates visual mathematical reasoning by analyzing LMMs’ knowledge acquisition and generalization, rather than relying only on end-to-end answers. Its evaluation finds persistent knowledge and multi-step reasoning challenges, while GPT-4o advances toward generalization and knowledge augmentation reduces insufficient-knowledge issues.

1 Introduction

  • GPT-4o achieves the best overall performance; most LMMs fare worse on multi-step problems, excel at calculation, and struggle with fine-grained visual measurement.Closed-source models and larger LMMs generally show stronger visual mathematical reasoning.Source: Figure 1 · p. 2, §1 · p. 3
  • Prior benchmarks rely mainly on end-to-end results, obscuring reasoning-process problems; WE-MATH argues evaluation should be based on knowledge concepts.The introduction asks whether a correct answer demonstrates accurate reasoning and whether a wrong answer indicates missing foundational knowledge.Source: §1 · p. 2
  • WE-MATH evaluates reasoning through knowledge concepts, decomposing composite-problem performance into solving constituent concepts and integrating them.It includes over 6.5K problems across 67 concepts and five granularity layers; 1.5K multi-concept problems are decomposed into sub-problems and assessed with IK, IG, CM, and RM. KCA supplies concept descriptions drawn from Wikipedia and textbooks to address IK.Source: §1 · pp. 2–3
  • Most LMMs exhibit Insufficient Knowledge (IK), while GPT-4o advances toward generalization and some models show Rote Memorization (RM), including G-LLaVA-13B at nearly 36% RM (Loose).Smaller models such as LLaVA-1.6-7B and DeepSeek-VL-1.3B have over 350 IK issues; the KCA strategy substantially reduces IK.Source: §1 · p. 3

2 WE-MATH

WE-MATH organizes visual math problems around textbook knowledge units and evaluates how LMMs use individual concepts to solve composite problems. It combines a hierarchical dataset, knowledge-based problem decomposition, four reasoning categories, and knowledge-concept augmentation.

2.1 Hierachial Structured Dataset Composition

  • The dataset’s hierarchy spans five math categories, 12 typical problem types, and 67 knowledge concepts, with 10–40 samples per terminal concept to balance domains.Concepts at the same level are independent, while concepts at different levels have logical relationships.Source: §2 · p. 3, §2.1 · p. 4
  • The 6.5K problems are manually concept-labeled by three experts, cross-validated, and checked for question–diagram consistency, answer correctness, and concept alignment.Problems are also labeled as One-Step, Two-Step, or Three-Step based on their solution steps.Source: §2.1 · p. 4

2.2 Knowledge based Reasoning Evaluation

  • WE-MATH decomposes each composite problem into one-step sub-problems by knowledge concept, enabling evaluation of both the original problem and its component reasoning.The number of concepts defines the problem’s step count; sub-problem conditions are built recursively to preserve logical coherence.Source: §2.2 · pp. 4–5
  • The four categories are Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM), with the capability hierarchy IK < IG < CM.IG means all one-step problems are correct but the composite problem is wrong; RM means one-step errors coexist with a correct composite answer.Source: §2.2 · pp. 5–6
  • A loose metric treats the two-step TFT and FTT cases as Complete Mastery rather than Rote Memorization to account for model instability.The paper also discusses the three-problem metric in Appendix C.Source: §2.2 · p. 6

2.3 Knowledge Concept Augmentation

  • WE-MATH provides 67 expert-reviewed knowledge-concept cards, distilled from Euclid’s Elements, Wikipedia, and textbooks, for models to use when answering problems.Experts refine each card using questions about its concept to extract relevant knowledge hints.Source: §2.3 · pp. 6–7

3 Experiment

Across WE-MATH experiments, LMM performance declines as problem complexity grows, with knowledge gaps, generalization, and fine-grained visual measurement emerging as major challenges.

3.1 Main Result

  • GPT-4o leads across problem complexities, but its accuracy falls from 72.84% on one-step to 43.64% on multi-step problems, reflecting the broader multi-step performance decline.Table 1 also reports accuracy on three-step problems and across one-step problem categories.Source: §3.1 · p. 7, Table 1 · p. 8
  • Larger text-decoder scales generally improve generalization, yet smaller Phi3-Vision and MiniCPM models are competitive; models struggle with measurement and conceptual understanding despite stronger calculation.The authors identify potential for parameter compression and emphasize optimizing visual representations.Source: §3.1 · pp. 8–9

3.2 Knowledge based Reasoning Analysis

  • All LMMs show Insufficient Knowledge (IK), especially smaller models, and this knowledge gap in one-step problems hinders progress on composite problems.The analysis treats addressing IK as important for progressing toward Inadequate Generalization (IG) and Complete Mastery (CM).Source: §3.2 · p. 9

3.3 Quantitative Analysis

  • Knowledge concept augmentation (KCA) consistently improves strict and loose performance and alleviates IK, but does not noticeably improve IG.Knowledge errors appear across over 45 of the 67 concepts; visual errors are second most common but concentrated, with over 38 concepts showing none.Source: Figure 10 · p. 10, §3.3 · p. 11, Figure 11 · p. 11

3.2 Knowledge based Reasoning Analysis

  • GPT-4o is markedly superior to LLaVA-NeXT-110B in IK (19.05%) but weakest in IG, suggesting its main challenge has shifted toward knowledge generalization.The authors argue that IG comparisons require a solid grasp of IK, reflecting the hierarchy IK < IG < CM.Source: §3.2 · pp. 9–10
  • On the loose metric, GPT-4o has SRM < 2%, while other LMMs remain near 25% SRM, and several models’ SRM varies substantially between strict and loose metrics.The authors interpret that variation as evidence of some one-step ability whose performance fluctuates with prompting templates and hyper-parameters.Source: §3.2 · p. 10

4 Related Work

Prior benchmarks evaluate mathematical reasoning across text and visual tasks, while broader LMM benchmarks span modalities and capabilities. WE-MATH is introduced to assess LMM reasoning across diverse mathematical categories.

Mathematical Reasoning Benchmarks

  • WE-MATH extends mathematical reasoning evaluation across a wide range of categories, following text-based benchmarks such as GSM8K [50] and MATH [51] and visual geometry benchmarks such as GeoQA [32], UniGeo [33], and Geometry3K [30].MathVerse [35] evaluates reasoning paths against reference answers.Source: §4 · p. 11

Benchmarks for Large Multimodal Model

  • LMM benchmarks have expanded from narrow visual question-answering and image-captioning tasks to multiple-choice commonsense and academic-domain evaluations, including MMBench [79], SEED-bench [80], and MMMU [54].MMStar [82] highlights a limitation: some evaluations can be answered without images, risking data leakage and inadequate measurement of logic and reasoning.Source: §4 · p. 11
  • LMM evaluation also spans 2D images, 3D point clouds, audio, video [99], and tool-using agents [101] [105].Understanding image implications can require multi-hop reasoning and theory of mind [84] [85] [87].Source: §4 · p. 11

5 Conclusion … B.1 Hierarchical Knowledge Structure

WE-MATH evaluates visual mathematical reasoning through knowledge-concept-based problems and fine-grained metrics, revealing LMMs’ knowledge and generalization challenges. The paper also describes the benchmark’s reliability focus, applications, ethical safeguards, and hierarchical structure.

5 Conclusion

  • WE-MATH evaluations reveal a negative correlation between solving steps and problem-specific performance, identify IK as LMMs’ greatest vulnerability, and show GPT-4o’s main challenge shifting from IK to IG.The paper identifies GPT-4o as the first LMM moving toward the next stage of knowledge generalization.Source: §5 · p. 12
  • WE-MATH covers 6.5K visual math problems across 5 layers and 67 knowledge concepts, decomposes composite problems into sub-problems, and introduces four-dimensional evaluation metrics.The metrics are Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM).Source: §5 · p. 12

A Broaden Impact

  • In a loose setting, GPT-4o’s RM metric is 1.07%, which the authors present as evidence of the possibility of a reliable and accurate model.The authors motivate WE-MATH partly by arguing that reliability matters alongside performance in specific fields, especially in fundamental scenarios.Source: A Broaden Impact · p. 21
  • WE-MATH’s IK, IG, and CM metrics assess model capabilities, its RM metric reflects reliability, and Scoreaverage quantifies overall performance.The decomposed solution process may also support interactive tasks, self-supervised learning, information extraction, and deployment in education and other fields.Source: A Broaden Impact · p. 21
  • WE-MATH uses elementary math problems sourced from publicly available test questions, textbooks, and professional websites, compensates annotators fairly, and restricts use to academic research.The authors state that construction followed legal and ethical guidelines, confirm the data license, and accept responsibility for rights violations.Source: A Broaden Impact · p. 21

B.1 Hierarchical Knowledge Structure

  • WE-MATH’s hierarchy has 5 levels, 99 nodes, and 67 leaf nodes, and uses knowledge concepts as dataset units to evaluate problem-solving mechanisms with scientific rigor and guidance for improvement.The authors say existing benchmarks provide only broad evaluation and improvement guidance.Source: §B.1 · p. 21

B.2 Knowledge-based Data Decomposition · B.3 Knowledge Concepts Augmentation · B.4 Details of Data Collection

WE-MATH decomposes multi-step problems through expert reasoning, augments knowledge-concept descriptions with visual content, and collects image-based geometry problems using defined domain and quality criteria.

B.2 Knowledge-based Data Decomposition

  • Experts extract key information for each sub-problem from the concepts’ reasoning paths, then use it to formulate sub-problems and refine answer options.Figures 14 and 15 illustrate the decomposition process with two-step and three-step examples.Source: §B.2 · pp. 21–22, Figure 14 · p. 25, Figure 15 · p. 25
  • The first sub-problem retains the original problem’s text and image conditions to support rigorous, independent sub-problems.Source: §B.2 · p. 22
  • For the second sub-problem, the first answer is added either to its image or its text condition; each problem uses only one approach, usually visual, so the final question matches the original.Text is used when information is extremely difficult to present in images.Source: §B.2 · p. 22

B.3 Knowledge Concepts Augmentation

  • Prompts concatenate textual descriptions, while each knowledge-concept description also includes visual content to support comprehension and possible future model enhancement.For Perimeter of Squares, the relation c=4a is given as an example where text alone is insufficient.Source: §B.3 · p. 26, Table 3 · p. 26

B.4 Details of Data Collection

  • Image-based geometry problems were selected from authoritative mathematics websites across five foundational primary-geometry domains: plane figures, solid figures, transformations and motion, position and direction, and measurement.Source: §B.4 · p. 26
  • Selected problems must involve multiple knowledge concepts and admit stepwise decomposition, have images consistent with the problem, and offer a unique correct answer with confusing distractors.Source: §B.4 · p. 26

B.5 Details of Data Statistics

WE-MATH’s questions are English multiple-choice items collected from public mathematics websites, with varied lengths and a format designed to support consistent, scalable assessment.

Question distribution

  • English questions average 25.81 words, with a maximum of 143; Figure 16 groups lengths above 80 as 81 for display.Table 4 presents key statistics for WE-MATH.Source: §B.5 · p. 27, Table 4 · p. 27, Figure 16 · p. 27
  • All questions are English multiple-choice items collected from public authoritative mathematics websites, and researchers converted initially non-multiple-choice problems.Source: §B.5 · p. 27

Advantages of Multiple-Choice Questions

  • Multiple-choice questions standardize assessment, enable objective grading and scalable evaluation, and use distractors to identify knowledge gaps and misconceptions.Source: §B.5 · p. 27

C More Details on the Metrics

The section distinguishes strict and loose rules for classifying correctly answered multi-step problems as CM or RM. It also explains the metrics’ logic as measuring progression from mastering individual concepts to applying them together.

C More Details on the Metrics

  • Under the strict metric, any incorrect sub-problem makes a correctly answered multi-step problem RM, while CM requires all sub-problems correct; under the loose metric, RM requires all sub-problems incorrect, so Scoreaverage is slightly higher.The strict three-step and loose three-step cases are illustrated in Figures 17 and 18, respectively.Source: C More Details on the Metrics · p. 28, Figure 17 · p. 28, Figure 18 · p. 28
  • The metrics treat IK, IG, and CM as logically related, with reasonable learning progressing from mastering each knowledge concept individually to applying them comprehensively.The authors characterize correctly solving a multi-step problem while answering its sub-problems incorrectly (RM) as unreasonable, motivating the four-dimensional metric.Source: C More Details on the Metrics · p. 28

D More Details on Experiment Setup … E More Details on Experiment Results

The setup evaluates the latest versions of selected LMMs, including several smaller models to test potential improvement under KCA, using model-appropriate inference configurations.

D.1 Details of the Evaluated Models

  • Models with 7B, 4.2B, and 1.3B parameters were included to explore whether KCA could significantly improve smaller models’ performance on Insufficient Knowledge (IK).The evaluation otherwise selected the latest model versions; Table 5 is described as listing their release dates and sources.Source: §D.1 · p. 29

D.2 Details of the Model Hyperparameters

  • Closed-source models with API access were run on CPUs, while open-source models were evaluated on a cluster of 8 NVIDIA A800-SXM4-80GB GPUs.The CPU inference typically completed within a day.Source: §D.2 · p. 29
  • Open-source model inference followed source-provided hyperparameters, or default settings when instructions were unavailable; Table 7 presents the generation parameters.The passage does not provide the parameter values.Source: §D.2 · p. 29, Table 7 · p. 30
  • Table 8 presents the architectures of the 17 LMMs evaluated on We-Math.The supplied passage gives no architecture-specific details.Source: Table 8 · p. 30

E.1 Details of Model Performance

GPT-4o leads WE-MATH performance overall and on several four-dimensional metrics, while category-level results reveal measurement-related weaknesses and variation across models.

E.1 Details of Model Performance

  • GPT-4o leads under both loose and strict metrics, LLaVA-NeXT-110B is the strongest open-source model, and InternVL-Chat-V1.5 and GLM-4V-9B surpass Qwen-VL-Max.Several smaller models also show outstanding performance, suggesting that training-method optimization may partly substitute for gains from increasing parameter count.Source: §E.1 · p. 31
  • GPT-4o has the fewest IK cases, while GPT-4o and GPT-4V show the highest IG exposure and lead on CM and RM; GPT-4o outperforms GPT-4V under the strict metric.IG exposure arises after IK issues are addressed, which the paper interprets as GPT-4 progressing to the next stage; CM counts correctly answered multi-step problems and their sub-questions.Source: §E.1 · p. 31
  • GPT-4o leads across knowledge concepts but struggles most with measurement tasks; some open-source models fare worse on unit understanding than angles and length, while InternVL-Chat-V1.5 and MiniCPM-LLaMA3-V 2.5 are more logically consistent.The category analysis covers 17 models across 67 concepts, using one-step problem questions.Source: §E.1 · p. 31, Figure 19 · p. 31, Figure 25 · p. 33

E.2 Specific Error Analysis · F Example Demonstration · F.1 Description of the Knowledge Concepts

The error analysis links more complex knowledge concepts to more reasoning errors, while the appendix provides examples of GPT-4o responses and descriptions of the benchmark’s knowledge concepts.

E.2 Specific Error Analysis

  • The analysis categorizes four typical errors and illustrates them with GPT-4o examples: visual error, reasoning error, knowledge error, and hallucination.The paper counts only the first error in a reasoning process because one logical error can derail a larger solution.Source: §E.2 · p. 36, Figure 36 · p. 36, Figure 37 · p. 37
  • Errors in multi-step problems often correspond to Insufficient Knowledge errors in one-step problems, and more knowledge concepts are associated with more errors and harder reasoning.As concept complexity increases, visual recognition errors and incorrect application of knowledge become more likely.Source: §E.2 · p. 36

F.1 Description of the Knowledge Concepts

  • The knowledge-concept descriptions cover unit understanding and conversion, angles and length, and basic figure transformations.These descriptions are shown in Figures 40–42.Source: §F.1 · p. 39, Figure 40 · p. 39, Figure 41 · p. 39
  • Other descriptions cover direction and position, solid-figure calculation and understanding, and plane-figure calculation.These descriptions are shown in Figures 43–46.Source: §F.1 · p. 39, Figure 43 · p. 40, Figure 44 · p. 41
  • The remaining descriptions address understanding plane figures, route maps, and correspondence between coordinates and positions.These descriptions are shown in Figures 47–49; Figure 5 also presents detailed knowledge-concept information.Source: §F.1 · p. 39, Figure 47 · p. 43, Figure 48 · p. 44

F.2 Data Sources of WE-MATH

The appendix catalogs WE-MATH data sources across five tables, partitioned into numbered ranges from Source 1 to Source 223.

F.2 Data Sources of WE-MATH

Related papers

Loading 2407.01284v1…