Source-linked AI summary
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, Honggang Zhang
TL;DR
Existing visual mathematics benchmarks emphasize end-to-end performance without adequately revealing how LMMs acquire and generalize knowledge during reasoning. WE-MATH addresses this by decomposing composite problems into concept-based sub-problems and applying a four-dimensional evaluation across 6.5K problems, 67 concepts, and five granularity layers. The evaluation finds that performance declines as solving steps increase, Insufficient Knowledge is widespread and reducible through knowledge augmentation, and GPT-4o has advanced from that issue toward Inadequate Generalization.
Problem
Existing benchmarks focus primarily on end-to-end results and do not adequately identify knowledge acquisition and generalization issues underlying LMMs’ visual mathematical reasoning.
Method
WE-MATH collects over 6.5K visual math problems across 67 knowledge concepts and five granularity layers, decomposes composite problems into sub-problems, and evaluates four knowledge-based reasoning categories.
Results
Most LMMs perform worse as problems require more solving steps; Insufficient Knowledge is widespread, knowledge augmentation reduces it, and GPT-4o’s main challenge shifts toward Inadequate Generalization.
Takeaways & Limitations
WE-MATH supports analyzing visual mathematical reasoning beyond final answers and distinguishes knowledge deficits, generalization gaps, mastery, and rote memorization in LMMs.
Abstract
from arXiv · showhide
Visual mathematical reasoning, as a fundamental visual reasoning ability, has received widespread attention from the Large Multimodal Models (LMMs) community. Existing benchmarks, such as MathVista and MathVerse, focus more on the result-oriented performance but neglect the underlying principles in knowledge acquisition and generalization. Inspired by human-like mathematical reasoning, we introduce WE-MATH, the first benchmark specifically designed to explore the problem-solving principles beyond end-to-end performance. We meticulously collect and categorize 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and five layers of knowledge granularity. We decompose composite problems into sub-problems according to the required knowledge concepts and introduce a novel four-dimensional metric, namely Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of existing LMMs in visual mathematical reasoning and reveal a negative correlation between solving steps and problem-specific performance. We confirm the IK issue of LMMs can be effectively improved via knowledge augmentation strategies. More notably, the primary challenge of GPT-4o has significantly transitioned from IK to IG, establishing it as the first LMM advancing towards the knowledge generalization stage. In contrast, other LMMs exhibit a marked inclination towards Rote Memorization - they correctly solve composite problems involving multiple knowledge concepts yet fail to answer sub-problems. We anticipate that WE-MATH will open new pathways for advancements in visual mathematical reasoning for LMMs. The WE-MATH data and evaluation code are available at https://github.com/We-Math/We-Math.
1 Introduction
Visual mathematical reasoning benchmarks have largely emphasized end-to-end answers, leaving LMMs’ knowledge acquisition and generalization processes insufficiently examined. WE-MATH addresses this gap by decomposing composite problems into concept-based sub-problems and evaluating reasoning through four knowledge-based categories.
- Motivation: Existing benchmarks mainly assess end-to-end results, making it difficult to identify inherent issues in LMM reasoning processes.The paper frames this limitation around whether correct answers reflect genuine reasoning and whether incorrect answers indicate missing foundational knowledge.
- Benchmark: WE-MATH contains over 6.5K visual math problems spanning 67 knowledge concepts and five layers of knowledge granularity.The benchmark is designed for comprehensive coverage of visual mathematical reasoning.
- Evaluation design: The benchmark decomposes composite problems into individual sub-problems, then evaluates whether models solve each concept and integrate the concepts into a final answer.This separates solving k concept-specific sub-problems from producing the composite answer.
- Evaluation design: WE-MATH introduces four reasoning categories: Insufficient Knowledge, Inadequate Generalization, Complete Mastery, and Rote Memorization.These dimensions assess different gaps in knowledge-based problem solving rather than relying only on final-answer accuracy.
- Findings: Most LMMs perform significantly worse on multi-step problems than one-step problems, while GPT-4o achieves the best overall performance across visual mathematics categories.The paper reports that more knowledge concepts are associated with greater problem difficulty and lower LMM performance.
- Findings: Most LMMs show Insufficient Knowledge, whereas GPT-4o’s main challenge shifts toward Inadequate Generalization and other models show marked Rote Memorization.The paper also reports that the proposed knowledge concept augmented strategy substantially reduces Insufficient Knowledge issues.
2 WE-MATH
WE-MATH structures visual mathematics around hierarchical textbook knowledge, decomposes composite problems into concept-based sub-problems, and evaluates reasoning through four outcome categories. It also augments models with knowledge concept cards to address reasoning weaknesses.
- Hierarchical Knowledge Structure: The benchmark uses a hierarchical, multi-category structure that preserves concept independence within levels and logical relations across levels.
- Knowledge Concept Augmentation: Knowledge concept augmentation supplies descriptions for 67 concepts, derived from Euclid’s Elements, Wikipedia, and textbooks, to support LMM reasoning.Experts summarize definitions and condense question-derived knowledge hints into concept cards.
- Hierarchical Knowledge Structure: WE-MATH organizes mathematics into five categories, 12 typical problems, and 67 knowledge concepts arranged as terminal nodes in a tree.Each terminal node contains 10–40 samples.
- Knowledge-based Data Decomposition: Expert annotators decompose composite problems into one-step sub-problems according to their required knowledge concepts and prerequisite conditions.The decomposition recursively concatenates preceding answers to maintain logical coherence.
- Knowledge-based Reasoning Evaluation: WE-MATH evaluates models by feeding both original multi-step problems and their sub-problems into LMMs, classifying outcomes as IK, IG, CM, or RM.IK indicates errors in sub-problems and the composite problem; IG indicates correct sub-problems but an incorrect composite answer; CM indicates all are correct; RM reverses that pattern.
- Knowledge-based Reasoning Evaluation: The reasoning hierarchy is IK < IG < CM because improving generalization can potentially shift models from IG to CM.The benchmark treats IG as preferable to IK because it reflects proficiency on individual concepts, while IK reflects difficulty with both individual and composite knowledge.
- Knowledge-based Reasoning Evaluation: WE-MATH introduces a loose metric alongside the strict RM criterion, treating TFT and FTT cases in two-step problems as CM under the loose setting.The final confidence score combines weighted IK, IG, and CM components, with α < β < 1, defaulting to α = 0.0 and β = 0.5.
3 Experiment
WE-MATH evaluates LMMs across problem complexity, visual-math categories, and hierarchical knowledge-based metrics. Results show declining multi-step performance, persistent knowledge and generalization weaknesses, and benefits from knowledge concept augmentation.
- Main Results: Most LMMs perform significantly worse on multi-step than one-step problems, with GPT-4o accuracy dropping from 72.84% to 43.64%.The number of knowledge concepts is positively correlated with difficulty and negatively correlated with performance.
- Main Results: Larger text-decoder parameter scales generally improve strict-metric performance and visual mathematical generalization.The comparison uses LMMs sharing the same language-model backbone.
- Main Results: LMMs generally excel at formula-based calculation but struggle with fine-grained visual measurement and conceptual understanding.Angle Measurement and Unit Conversion are recurring weaknesses, while calculation categories are comparatively stronger.
- Knowledge-based Reasoning Analysis: Insufficient Knowledge is the greatest vulnerability, especially for smaller models, and knowledge gaps hinder reasoning over composite problems.The evaluation reports four dimensions: Insufficient Knowledge, Inadequate Generalization, Complete Mastery, and Rote Memorization.
- Knowledge-based Reasoning Analysis: GPT-4o’s main challenge shifts from IK toward IG, while other LMMs remain closer to the IK stage.GPT-4o is reported as 19.05% superior to LLaVA-NeXT-110B on IK, suggesting conversion of some IK into IG.
- Quantitative Analysis: KCA consistently improves strict and loose performance and significantly alleviates IK, but does not noticeably improve IG.The authors associate this difference with knowledge descriptions addressing reasoning-knowledge gaps rather than generalization ability.
4 Related Work
Prior mathematical-reasoning benchmarks evaluate text-based and multimodal capabilities across increasingly broad tasks. The related work motivates more comprehensive assessment of large foundational models and visual reasoning.
- Mathematical Reasoning Benchmarks: Early mathematical benchmarks evaluated operation-based reasoning in word problems and later expanded to text problems across multiple difficulty levels.Examples include MathQA, GSM8K, and MATH.
- Benchmarks for Large Multimodal Model: Large multimodal-model benchmarks emerged from text-only evaluations to address the visual reasoning capabilities of increasingly capable foundational models.The visual benchmark literature initially focused on narrower evaluation settings.
5 Conclusion
WE-MATH is presented as a benchmark for analyzing LMM reasoning principles in visual mathematics through hierarchical knowledge concepts and fine-grained metrics. It evaluates existing models, identifies distinct reasoning vulnerabilities, and offers guidance toward more human-like reasoning.
- WE-MATH analyzes LMMs’ visual mathematical reasoning using 6.5K problems organized across five layers and 67 knowledge concepts.
- The benchmark decomposes composite problems into sub-problems and uses IK, IG, CM, and RM for fine-grained evaluation.
- Evaluations reveal a negative correlation between solving steps and problem-specific performance, while identifying IK as the greatest vulnerability of LMMs.
- GPT-4o’s main challenge shifts from IK to IG, marking it as the first LMM moving toward the knowledge-generalization stage.
- WE-MATH’s RM metric captures cases where models solve complex problems but fail on sub-problems, reflecting concerns about reasoning reliability.
B.2 Knowledge-based Data Decomposition
WE-MATH constructs knowledge-based decompositions of multi-step visual math problems by extracting expert reasoning information and formulating logically related sub-problems. Its dataset uses hierarchical concepts, visual-textual prompts, and explicit selection criteria.
- Figures 14 and 15 illustrate the Knowledge-based Data Decomposition process for multi-step problems.
- Educational experts extract key information for each sub-problem from the reasoning path associated with the concepts in a multi-step problem.
- The Decompose stage uses extracted information to formulate sub-problems, refine options, and complete the multi-step decomposition.
- The first sub-problem retains the original text condition and image, while later sub-problems receive the preceding answer as a visual or textual condition.
- Only one answer-injection mode is used within a decomposed problem, ensuring the final sub-problem matches the original multi-step question.
- Problems are selected when they involve multiple concepts, consistent text and images, unique answers, and confusing distractors.
B.5 Details of Data Statistics
WE-MATH consists of English multiple-choice questions collected from authoritative public mathematics sources, with statistics describing question length and design choices supporting standardized, reliable assessment.
- Figure 16 shows the distribution of question word counts, grouping questions longer than 80 words as 81 for visualization.
- WE-MATH questions average 25.81 English words, with a maximum length of 143 words.
- All WE-MATH problems use multiple-choice format, including questions manually converted from non-multiple-choice sources.
- Multiple-choice questions standardize assessment, enable objective grading, and support rapid scalable evaluation.
- Carefully designed distractors help identify specific knowledge gaps and common misconceptions.
C More Details on the Metrics
WE-MATH distinguishes model reasoning patterns with strict and loose criteria for CM and RM, grounded in the relationship among IK, IG, and complete mastery. The metric design reflects a progression from individual concept mastery to integrated application.
- Under the strict metric, a correctly answered multi-step problem is CM only when all corresponding sub-problems are correct; otherwise it is RM.
- Under the loose metric, a correctly answered multi-step problem is classified as RM only when all sub-problems are answered incorrectly.
- The metric framework treats IK, IG, and CM as logically related stages in progressing from individual knowledge concepts to comprehensive application.
- RM identifies the unreasonable pattern of solving a multi-step problem correctly while failing its corresponding sub-problems.
D More Details on Experiment Setup
The evaluation includes recent models across parameter scales, using model-specific generation settings and additional smaller models to examine knowledge augmentation.
- Model Selection: The study evaluates latest model versions, including models with 7B, 4.2B, and 1.3B parameters.The smaller models were included to investigate whether knowledge augmentation could significantly improve their performance.
- Inference Setup: Closed-source models are run through API inference, while open-source models are evaluated on eight NVIDIA A800-SXM4-80GB GPUs.Closed-source inference typically completes within a day; open-source models follow source-recommended settings or defaults.
- Inference Setup: Open-source models use the generation parameters specified in Table 7.The supplied materials identify Table 7 as the source of these parameters.
- Inference Setup: LongVA uses deterministic generation with temperature 0, one beam, sampling disabled, and a maximum of 1024 new tokens.These settings are reported explicitly for LongVA.
- Model Selection: The benchmark evaluates 17 LMM architectures.Table 8 presents the model architectures evaluated on WE-MATH.
E.1 Details of Model Performance
WE-MATH compares LMMs across overall scores, four diagnostic metrics, and 67 knowledge concepts. GPT-4o leads overall and diagnostically, while model performance varies across concepts and model scales.
- Overall Performance: GPT-4o leads WE-MATH under both loose and strict Scoreaverage metrics, while LLaVA-NeXT-110B is the strongest open-source model.InternVL-Chat-V1.5 and GLM-4V-9B surpass Qwen-VL-Max, and several smaller models also show strong performance.
- Four-Dimensional Metrics: GPT-4o has the fewest Insufficient Knowledge instances and the highest exposure to Inadequate Generalization alongside GPT-4V.The paper interprets this combination as evidence that GPT-4 is progressing beyond insufficient knowledge toward generalization.
- Four-Dimensional Metrics: GPT-4o and GPT-4V lead Complete Mastery and Rote Memorization, with GPT-4o outperforming GPT-4V under the strict metric.Complete Mastery counts correctly answered multi-step problems together with their corresponding sub-questions.
- Concept-Level Performance: Across 67 knowledge concepts, GPT-4o consistently leads overall but shows its main weakness on measurement-related tasks.The detailed concept-level results are based on one-step problem questions.
E.2 Specific Error Analysis
The error analysis links multi-step failures to one-step knowledge errors and examines visual, reasoning, knowledge, and hallucination errors through examples and concept descriptions.
- Error Types: The analysis distinguishes four typical error types: Visual Error, Reason Error, Knowledge Error, and Hallucination.Table 9 provides detailed descriptions, while Figures 36–39 illustrate GPT-4o examples.
- Error Correspondence: Multi-step Insufficient Knowledge errors often correspond to errors in one-step problems, supporting decomposition by knowledge concept.The analysis uses this correspondence to associate error types with specific concepts.
- Error Correspondence: More knowledge concepts are positively correlated with errors in the reasoning process.The passage also relates increasing conceptual complexity to greater problem difficulty.
- Knowledge Concepts: The benchmark describes knowledge concepts through figures covering units, angles, transformations, directions, positions, solid figures, plane figures, routes, and coordinates.These descriptions provide concept-specific reference material for the benchmark’s mathematical tasks.
- Knowledge Concepts: Position descriptions combine a reference point with direction, angle, and distance.The supplied concept description gives examples such as locating a hospital relative to a library.
F.2 Data Sources of WE-MATH
WE-MATH draws its problems from publicly authoritative mathematics websites and documents the sources in five source-list tables.
- Data Sources: The benchmark’s data sources are documented in Tables 10–14.The tables are divided into five parts covering source ranges 1–223.