Source-linked AI summary
Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, Hongsheng Li
TL;DR
Existing multimodal mathematical benchmarks have limited diversity and subject coverage, making it difficult to comprehensively assess visual mathematical reasoning. The paper constructs MATH-V, a curated benchmark of competition problems organized across disciplines and difficulty levels, and finds a substantial gap between current LMMs and human performance. Its categorization also supports detailed error analysis of LMMs.
Problem
Existing visual-mathematical benchmarks have limited diversity of questions and breadth of mathematical subjects.
Method
The paper constructs MATH-V from 3,040 competition problems, categorizing them across 16 mathematical disciplines and 5 difficulty levels after cross-validation and expert filtering.
Results
MATH-V reveals a substantial gap between human performance and current multimodal models, with respective scores of 75.66 and 22.76.
Takeaways & Limitations
MATH-V provides a more comprehensive evaluation of multimodal mathematical reasoning and enables detailed error analysis of current LMMs.
Takeaways & Limitations
MATH-V contains only English mathematics problems with abstract images and excludes multimodal reasoning problems in subjects such as physics and chemistry.
Abstract
from arXiv · showhide
Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io
1 Introduction
Existing visual-mathematical reasoning benchmarks suggest near-human LMM performance but have limited subject diversity and problem coverage. MATH-V addresses these limitations with a curated, broad benchmark and reveals a substantial remaining gap between LMMs and humans.
- Limitations of Existing Benchmarks: MathVista reports 57.6 for current models versus 60.3 for humans, although some subtasks surpass average-human performance.These results motivate questioning whether existing benchmark scores reflect human-comparable mathematical reasoning.
- Limitations of Existing Benchmarks: MathVista’s math-question sources cover a narrow range of subjects, mainly plane geometry, tables, and charts.Newly collected datasets add diversity, but problems remain sparse within individual subjects.
- MATH-V Benchmark: MATH-V contains 3,040 visual-context problems from 19 math competitions, cross-validated and filtered by multiple expert annotators.The dataset includes 1,532 open-ended and 1,508 multiple-choice problems.
- MATH-V Benchmark: MATH-V classifies problems across 5 difficulty levels and 16 mathematical disciplines, with human verification of the classification.The categorization is intended to support analysis across difficulty levels and subject areas.
- Experimental Findings: 22.76 and 75.66 are the respective scores of LMMs and humans on MATH-V, indicating a substantial visual mathematical-reasoning gap.GPT-4V and Gemini perform better than publicly trained models but still struggle with invariant geometric properties under continuous deformations.
- Experimental Findings: Fine-grained categorization enables comprehensive error analysis of current LMMs and provides insights for improvement.The benchmark is designed to evaluate mathematical reasoning across varied visual problems rather than a narrow subject range.
2 Related Works
Prior work includes multimodal benchmarks, math-specific models, and attempts to extend text-only models to multimodal reasoning. These systems show strong results, but multimodal mathematical reasoning remains an active benchmarked capability.
- Multimodal Mathematical Benchmarks: Existing multimodal benchmarks primarily focus on visual recognition, while multimodal mathematical reasoning benchmarks are less common.Early benchmarks such as the cited works and GeoQA+ are described as limited in scope.
- Math-Specific Models: Math-specific LLMs such as Llemma and Mathcoder are competitive with general language models on text-only mathematical reasoning at smaller scale.The passage contrasts text-only mathematical competence with multimodal reasoning capabilities.
- Math-Specific Models: AlphaGeometry performs impressively on challenging geometry problems but cannot process images and relies on textual image descriptions.This illustrates a distinction between geometry reasoning and direct visual mathematical reasoning.
- Large Multimodal Models: Models including MiniGPT-4, LLaMA-Adapter, and LLaVA attempted to construct multimodal systems from text-only models.These efforts preceded broader large multimodal models such as GPT-4V and Gemini.
- Large Multimodal Models: Large multimodal models including GPT-4V, Gemini, Qwen-VL, InternLM-XComposer-VL, and SPHINX demonstrated strong performance on MMMU and MathVista.The related work situates MATH-V among these multimodal evaluation efforts.
3 Dataset
MATH-V is a curated visual mathematical reasoning benchmark built from real competitions, spanning diverse subjects and difficulty levels. Its collection and curation processes produce a larger, structured dataset for evaluating multimodal models.
- Task design: The benchmark emphasizes expert-level visual perception and deliberate reasoning with subject-specific knowledge across heterogeneous image types.
- Overview: 3,040 questions form the MATH-V benchmark, with a smaller 304-question test set for quick evaluation.
- Data collection: The collection process gathers competition problems with frequent visual inputs while considering copyright, licensing, and data-contamination concerns.
- Data curation: A four-stage process with 10 annotators verifies text-image alignment, removes invalid or duplicated items, and categorizes the remaining questions.
- Comparison with existing benchmarks: Compared with MathVista and MMMU-Math, MATH-V uses newly collected competition data, longer questions than MathVista, and a substantially larger mathematics set than MMMU-Math.MATH-V questions average 42.3 words versus 15.6 for MathVista, while its dataset is six times larger than MMMU-Math.
4 Experiments
Experiments show that MATH-V remains difficult for current multimodal models, with performance varying substantially across subjects, model types, prompting strategies, and error categories.
- Experimental Setup: Experiments evaluate open- and closed-source LLMs and LMMs in zero-shot settings, with and without Chain-of-Thought prompting.The study also includes human-performance evaluation.
- Challenging Nature of MATH-V: 22.76% overall accuracy was achieved by GPT-4V, while open-source models all remained below 15%.GPT-4V exceeded random chance, which was 7.17%.
- Comparison Among Subjects: 35.7% arithmetic accuracy contrasted with GPT-4V’s below-15% accuracy in counting, revealing strong subject-level variation.The results indicate greater difficulty on problems requiring more complex visual recognition and reasoning.
- Closed- and Open-Source Models: 22.76% overall accuracy for GPT-4V exceeded InternLM-XComposer2-VL’s 14.54%, illustrating a closed-source advantage on MATH-V.The paper notes that the opposite ordering appears on MathVista.
- CoT Prompting: CoT increased GPT-4V accuracy from 22.76% to 23.98%, but decreased Gemini Pro and Qwen-VL-Max accuracy by 3.32% and 2.96%.The authors interpret this inconsistency as evidence that step-by-step reasoning capabilities remain immature.
- GPT-4V Error Analysis: 42.2% of GPT-4V errors were reasoning errors and 31.9% were vision-recognition errors, whereas calculation errors accounted for 1.3%.The calculation errors involved symbolic rather than numerical computation.
- Caption-Based Analysis: GPT-4 with GPT-4V-generated image captions reached 13.10 performance, comparable to SPHNIX and Qwen-VL.Caption quality affected visual perception and downstream mathematical reasoning across the 16 subjects.
5 Conclusion
The paper introduces MATH-V as a benchmark for multimodal mathematical reasoning and reports a substantial performance gap between current models and humans.
- Conclusion: MATH-V contains 3,040 challenging mathematical reasoning problems, each accompanied by images and texts.The dataset is intended to evaluate multimodal models’ mathematical reasoning abilities.
- Conclusion: MATH-V reveals a significant gap between human performance and current multimodal models while supporting systematic evaluation for future studies.The authors evaluated both open-source and closed-source models.
6 Limitations
MATH-V is limited to English mathematical problems with abstract images, excluding other languages, scientific subjects, and real-world photographs.
- Language Scope: All MATH-V problems are in English, so the dataset cannot evaluate reasoning in languages other than English.The authors identify this as a limitation requiring further investigation.
- Subject Scope: The benchmark includes only mathematics, excluding multimodal reasoning problems in subjects such as physics and chemistry.This constrains the range of domains assessed.
- Visual Scope: MATH-V uses abstract images rather than pictures taken in real life.The authors identify this as another limitation warranting further investigation.
7 Ethics statement
The ethics statement describes MATH-V’s public-source provenance, fair-use rationale, intended research use, and MIT licensing, while acknowledging misuse risks.
- Legal Compliance: MATH-V problems come from publicly available sources associated with MAA, KMA, and UKMT.The paper discusses legal compliance and fair-use considerations.
- Legal Compliance: The authors invoke Fair Use §107 for MATH-V’s scholarly and research use.They discuss purpose, commercial character, and effects on the potential market.
- Intended Usage: The dataset and reproduction code are intended for evaluating LMM mathematical reasoning, not for training models to cheat.The paper acknowledges that misuse remains a risk.
- License: MATH-V data and open-source code are released under an MIT license.
- Testmini Validation: Testmini mirrors full-test performance trends closely, with Qwen-VL-Plus differing by 0.19% and GPT-4V by -0.39%.Even the largest reported deviations remained within 2%.
B Evaluation Details
The evaluation uses prompts and model sources designed for multimodal mathematical reasoning, while comparisons expose limited question diversity and subject coverage in MathVista.
- B Evaluation Details: Evaluation prompts are designed to keep generated responses consistent with both the input question and image.A separate caption-generation prompt asks models to describe diagrams in complete detail for math problem-solving.
- B Evaluation Details: Models evaluated come from OpenAI, Google, Alibaba, and Hugging Face, with some accessed through platform APIs.
- C.1 Comparing with MathVista: MathVista contains repeated and template-like questions, including 310 questions containing “What is the age gap between” and 400 object-subtraction questions.After removing identical stems, 4,740 of 6,141 questions remain.
- C.1 Comparing with MathVista: MathVista’s math-question sources remain concentrated in a narrow range of subjects, while newer datasets are diverse but sparse per subject.The cited discussion contrasts limited coverage with the need for broader mathematical evaluation.
D More Dataset Details
MATH-V questions are diverse in length, difficulty, visual form, and mathematical subject, with difficulty assigned from competition grades and manually corrected.
- D More Dataset Details: MATH-V questions average 42.3 words, range from 5 to 280 words, and have a near-normal word-count distribution.The median and average word counts are close, indicating little skew.
- Difficulty Levels: Difficulty levels are initially mapped from competition grades 1–12 into five groups, then manually adjusted for within-exam difficulty variation.Grades 1–2 form level 1; grades 3–6 level 2; grades 7–8 level 3; grades 9–10 level 4; and grades 11+ level 5.
- Difficulty Levels: MATH-V and MATH have fewest questions at level 1 and similar numbers at other levels, while model accuracy decreases as difficulty increases.
- D.4 Introduction of Subjects: The dataset spans subjects including algebra, analytic geometry, arithmetic, combinatorial geometry, combinatorics, counting, logic, statistics, and topology.Visual patterns vary by subject: statistics emphasizes charts and tables, whereas arithmetic and counting use more colorful or concrete imagery.
- D.4 Introduction of Subjects: Arithmetic studies numbers and basic operations, combinatorics studies arrangements and selections, counting enumerates objects, and combinatorial geometry studies geometric arrangements combinatorially.
- D.4 Introduction of Subjects: Logic concerns valid argument structure, statistics concerns data analysis and inference, topology concerns properties preserved under continuous transformations, and analytic geometry connects coordinates, algebra, and shapes.
E.1 Reasoning Error (42.2%)
Reasoning errors arise when GPT-4V applies mathematical or logical steps incorrectly.
- E.1 Reasoning Error (42.2%): GPT-4V makes reasoning errors by substituting an unknown with a known value or hypothesizing illogical relationships between train-wagon numbers.
E.2 Vision Recognition Error (31.9%)
Vision-recognition errors arise when GPT-4V misinterprets or hallucinates visual information.
- E.2 Vision Recognition Error (31.9%): GPT-4V fails to read a simple bar chart, misidentifies triangle-side relationships, and hallucinates a balance absent from the image.
E.3 Knowledge Error (15.1%)
Knowledge errors occur when GPT-4V lacks the information needed to answer and produces outdated or incorrect information.
- Knowledge errors reflect missing, outdated, or incorrect information that prevents GPT-4V from solving some problems.Figures 40 and 41 provide examples of these failures.
E.4 Question Misunderstood Error (6.9%)
Question-misunderstanding errors arise when GPT-4V misinterprets the user’s query or assumes information that the question does not provide. The surrounding examples also distinguish refusal, calculation, and image-caption failures.
- GPT-4V sometimes misinterprets the query’s intent, producing an incorrect answer.One example shows the model incorrectly assuming that a key below corresponds to the lock above without textual support.
- Reject-to-answer errors occur when GPT-4V cannot generate a response or chooses not to answer.The examples attribute these refusals to various reasons.
- Calculation errors in the analysis are symbolic mistakes that undermine GPT-4V’s reliability in symbolic computation.The cited examples involve simple symbolic computational errors.
- Image-caption quality varies across subjects: captions are accurate or mildly hallucinated in some examples but usually inaccurate in others.The comparison includes GPT-4 alongside GPT-4V, Gemini Pro, and Qwen-VL-Max.