Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset

Benchmark limitations

• MathVista reaches 57.6 versus humans’ 60.3, yet its questions are often simple and repetitive, limiting assessment of deep mathematical reasoning.Its visual questions are concentrated in a few types, such as object counting, reference-based length measurement, and clock-based time inquiry.Source: §1 · p. 2
• Existing Math-QA-derived benchmarks cover a narrow range of subjects, mainly plane geometry, tables, and charts, while newly collected datasets remain sparse per subject.These limitations motivate broader coverage across mathematical disciplines and difficulty levels.Source: §1 · p. 2

Dataset contribution

• MATH-Vision contains 3,040 expert-validated problems from 19 math competitions, balanced across 1,532 open-ended and 1,508 multiple-choice questions.The dataset spans 12 grades, five difficulty levels, and 16 mathematical disciplines, with unique correct answers ensured through expert filtering.Source: §1 · pp. 2–3

Analysis contribution

• MATH-V’s fine-grained difficulty and subject classification supports comprehensive error analysis of current LMMs and provides insights for improving visual mathematical reasoning.The classification is human-verified to improve accuracy and reliability across levels and subject areas.Source: §1 · pp. 2–3

Empirical findings

• MATH-V scores reveal a substantial visual-mathematical reasoning gap between LMMs and humans, at 22.76 versus 75.66, while top models still fail on geometric invariants under deformation.GPT-4V and Gemini outperform publicly trained models and handle basic computation well, but are not yet comparable to average humans.Source: §1 · p. 2

2.1 Mathematical Reasoning Benchmarks

• MATH-Vision covers 16 representative subjects with clear difficulty levels and problems rigorous in both mathematical reasoning and visual recognition, unlike narrower prior benchmarks.Early benchmarks focus mainly on plane geometry; MMMU uses multiple-choice questions with little mathematical content, while MathVista lacks detailed subdomain classification and emphasizes visual abilities.Source: §2.1 · p. 3

2.2 LMMs for Math

• Math-specific LLMs such as Llemma and Mathcoder are competitive with GPT-4 and PaLM-2 on text-only reasoning at smaller scale, while AlphaGeometry solves challenging geometry without processing images.Current multimodal math models, including G-LLaVA, UniMath, MatCha, and UniChart, mainly target plane geometry or charts and lag behind general multimodal models such as GPT-4V.Source: §2.2 · p. 3

2.3 Multimodal Foundation Models

• GPT-4V performs considerably better than other models on MATH-Vision, indicating that open-source multimodal models still lack generalization, especially on questions unlike their fine-tuning data.The broader multimodal-model landscape includes MiniGPT-4, LLaMA-Adapter, LLaVa, Gemini, Qwen-VL, InternLM-XComposer-VL, and SPHINX, with strong results on MMMU and MathVista.Source: §2.3 · p. 3

3.1 Overview

• MATH-V contains 3,040 visual-context mathematics questions across 16 subjects and 5 difficulty levels, with a 304-question test set and problems collected from 19 competitions.The benchmark targets heterogeneous images, expert-level visual perception, subject-specific knowledge, and deliberate multimodal mathematical reasoning.Source: §3.1 · p. 4

3.2 Data Collection

• The two-stage collection process selected 19 competitions emphasizing frequent visual inputs and varied grade levels, then filtered online sources to obtain 9,000 questions, including 3,500 with image context while observing licensing and contamination concerns.PDF problems were processed with the Mathpix API to extract text and images, and sources lacking readily available answers were prioritized.Source: §3.2 · pp. 4–5
• Four-stage curation with 10 annotators removed misaligned, incomplete, duplicated, private, or offensive items, yielding 3,040 final data; annotators and models then assigned subjects by majority vote and difficulty from competition grade requirements.The process retained 3,352 items after initial validation, used lexical overlap and Levenshtein distance for duplicate detection, and consolidated subject labels through three expert annotators.Source: §3.2 · p. 5

3.3 Comparison with Existing Benchmarks

• MATH-V’s questions average 42.3 words versus MathVista’s 15.6, provide 18 fine-grained subject categorizations and five difficulty levels, and contain six times as many problems as MMMU-Math, with 190 versus 63 problems per subject on average.Unlike MMMU-Math’s college-level, domain-specific coverage, MATH-V focuses on elementary- to high-school mathematics with little specialized domain knowledge; MathVista has seven subject categories, three difficulty levels, and 37.7% not-applicable difficulty labels.Source: §3.3 · pp. 5–6

4.1 Experimental Setup

• Experiments evaluate LLMs with GPT-4V-generated captions, open-source and closed-source LMMs, using zero-shot prompts with and without CoT and a 30-student human baseline.Default prompts were used when available; otherwise, prompt engineering selected the strongest zero-shot prompt.Source: §4.1 · pp. 6–7

4.2 Experimental Results

• GPT-4V achieves the highest overall accuracy at 22.76%, while all open-source models remain below 15%, confirming MATH-V’s difficulty despite performance above 7.17% random chance.LLaVA-v1.5-7B exceeds random chance by 1.35%.Source: §4.2 · p. 7
• Across subjects, GPT-4V scores below 15% in counting but reaches 35.7% in arithmetic, revealing large differences in models’ mathematical and visual-reasoning strengths.Gemini Pro scores 7.5% in counting and 20.7% in arithmetic.Source: §4.2 · p. 7
• Closed-source GPT-4V reaches 22.76% overall accuracy and leads algebra at 27.3%, analytic geometry at 32.1%, and arithmetic at 35.7%, versus 14.54% for InternLM-XComposer2-VL.On MathVista, InternLM-XComposer2-VL-7B instead scores 7.7% and 12.4% higher than GPT-4V and Gemini Pro, respectively.Source: §4.2 · p. 7
• CoT prompting raises GPT-4V from 22.76% to 23.98% but lowers Gemini Pro by 3.32% and Qwen-VL-Max by 2.96%, showing inconsistent benefits.The results suggest that LMM step-by-step reasoning capabilities remain insufficiently mature to exploit CoT reliably.Source: §4.2 · p. 7

4.3 Error Analysis

• Among 232 GPT-4V errors, reasoning errors account for 42.2%, vision-recognition errors 31.9%, knowledge errors 15.1%, and calculation errors 1.3%.All calculation errors involved symbolic rather than numerical computation, while GPT-4 with image captions reaches 13.10 and can surpass GPT-4V in combinatorics and counting when descriptions are more detailed.Source: §4.3 · p. 8, Figure 4 · p. 8

• MATH-V comprises 3,040 challenging mathematical reasoning problems, each accompanied by images and texts.Source: §5 · pp. 8–9
• Evaluations of open-source and closed-source models systematically reveal a significant gap between human performance and current multimodal models on multimodal reasoning.Source: §5 · p. 9
• The benchmark facilitates more effective evaluation in future multimodal reasoning studies.Source: §5 · p. 9

• The benchmark contains only English math problems, preventing evaluation of reasoning in other languages.Source: §6 · p. 9
• It excludes multimodal reasoning in subjects such as physics and chemistry and uses only abstract, non-real-life images.The authors identify these limitations as important directions for further investigation.Source: §6 · p. 9

7 Ethics statement

• MATH-V questions come from publicly available sources, including problems created by MAA, KMA, and UKMT.The dataset contains free test questions published on the official KMA and UKMT websites, while the paper discusses MAA problem IP history.Source: §7 · p. 9
• The authors state that MATH-V follows Fair Use §107 for scholarship and research, considering nonprofit educational purpose and effects on the copyrighted work’s market.They cite the fair-use factors concerning purpose and character of use and potential market effects.Source: §7 · p. 9
• MATH-V and its reproduction code will be publicly available, with the dataset and open-source code released under an MIT license.The dataset is intended to help researchers evaluate LMM mathematical reasoning, although the authors acknowledge a risk of use for cheating and accept responsibility for rights violations.Source: §7 · p. 9

A Testmini

• Testmini performance tracks the full test set closely across mathematical subjects, supporting its use for preliminary evaluation and model comparison.The authors report parallel trends across the two datasets and describe testmini as indicative of full-test performance.Source: A Testmini · p. 13
• 0.19% is the Qwen-VL-Plus full-versus-testmini performance delta, while GPT4V differs by -0.39%.These negligible differences support the subset’s similarity to the full test set.Source: A Testmini · p. 13
• Even the largest reported deviations remain below 2%: 1.98% for LLaVA-v1.5-13B and 1.97% for ShareGPT4V-7B.The authors interpret these deviations as evidence that testmini reliably reflects full-test characteristics.Source: A Testmini · p. 13

B Evaluation Details

• MATH-V experiments are conducted on A100 80G GPUs.Source: B Evaluation Details · p. 13

B.1 Prompts for Response Generation

• Response-generation prompts are designed to make outputs consistent with both the input question and image, with the full prompts provided in Table 5.Source: §B.1 · p. 13

B.2 Prompt for Caption Generation

• Caption generation asks models to describe a math-problem diagram in complete detail so the description can replace the diagram for problem-solving.Source: §B.2 · p. 13

B.3 Models

• The evaluated models come from OpenAI, Google, Alibaba, and Hugging Face, with the first three accessed through their respective APIs.Source: §B.3 · p. 13

C.1 Comparing with MathVista

• After removing identical stem texts, MathVista’s 6,141 questions reduce to 4,740, while repeated templates include 310 age-gap questions and 400 similar geometry-recognition questions.Source: §C.1 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15
• Compared with MathVista’s predominantly abstract-scene and concise function-plot questions, MATH-V includes longer analytic-geometry problems with concepts such as mutual symmetry and iterative functions, plus topology and graph theory categories absent from MathVista.Source: §C.1 · p. 13

C.2 Comparing with MMMU

• MMMU questions target advanced college-level concepts such as Kruskal’s algorithm, alternating group structures, and DFS Spanning Tree, whereas MATH-V primarily presents puzzles solvable by middle- and high-school students.Source: §C.2 · pp. 13, 16

D.1 Question Distribution

• More than 98.75% of MATH-V questions contain fewer than 100 words, with an average of 42.3, a median close to the average, and a 5–280-word range.The word-count distribution is near normal and is illustrated in Figure 16.Source: §D.1 · p. 16, Figure 16 · p. 16

D.2 Division of Levels

• Difficulty levels are initially assigned from competition grades 1–12 and then manually corrected when within-exam difficulty overlaps make grade-based divisions unreasonable.Grades 1–2 map to level 1, 3–6 to level 2, 7–8 to level 3, 9–10 to level 4, and 11+ to level 5.Source: §D.2 · p. 16
• MATH-V and MATH have the fewest questions at level 1 and similar counts at other levels, while model accuracy decreases as difficulty level increases.The dataset-level comparison appears in Figure 17, and level-wise model accuracy is referenced through Figure 1.b.Source: §D.2 · p. 16, Figure 17 · p. 16

D.3 Images by Subjects

• MATH-V images span 16 subjects, with category-specific visual patterns ranging from charts and tables in Statistics to colorful primary-school imagery and abstract Analytic Geometry diagrams.Examples cover Algebra, geometry variants, Arithmetic, Combinatorics, Counting, Logic, Graph Theory, Statistics, Topology, and Transformation Geometry.Source: §D.3 · p. 17

D.4 Introduction of Subjects

• The subject introductions define mathematical domains through their characteristic objects and operations, such as algebraic manipulation, counting and arrangement, geometric measurement, graph connectivity, and shape transformations.These definitions frame the dataset’s categories across algebra, combinatorics, metric geometry, graph theory, and transformation geometry.Source: §D.4 · pp. 17–18
• The subject taxonomy covers foundational and advanced areas, including coordinate-based Analytic Geometry, discrete Combinatorial Geometry, formal Logic, data-oriented Statistics, topology under continuous transformations, and three-dimensional Solid Geometry.The taxonomy also includes Arithmetic, Combinatorics, Counting, Descriptive Geometry, Graph Theory, metric geometry, and Transformation Geometry.Source: §D.4 · pp. 17–18

E Error Examples of GPT-4V

• GPT-4V errors are categorized into reasoning, vision recognition, knowledge, question misunderstanding, refusal to answer, and calculation errors.The section presents detailed examples for these six error types.Source: E Error Examples of GPT-4V · p. 18

E.1 Reasoning Error (42.2%)

• GPT-4V makes reasoning errors by incorrectly substituting an unknown variable with a known value or hypothesizing illogical relationships between train wagon numbers.These mistakes lead to incorrect conclusions.Source: §E.1 · p. 18

E.2 Vision Recognition Error (31.9%)

• Vision recognition errors include failing to read a simple bar chart, mistakenly placing 3, 6, and y on the same side of a triangle, and hallucinating a balance absent from the image.These examples reflect difficulties interpreting visual data and grounding responses in visual evidence.Source: §E.2 · p. 19

• Knowledge errors arise when GPT-4V lacks the necessary information to answer a question.Source: §E.3 · p. 19
• These errors can involve providing information that is outdated or incorrect.Source: §E.3 · p. 19
• Figures 40 and 41 show cases where GPT-4V did not know how to solve the problems.Source: §E.3 · p. 19

E.4 Question Misunderstood Error (6.9%)

• GPT-4V makes errors when it misinterprets a user’s query, including failing to grasp its intent and mistakenly assuming that the key below corresponds to the lock above.Figure 42 demonstrates a failure to grasp the query’s intent, while Figure 43 shows the mistaken key-lock correspondence despite the question not providing this information.Source: §E.4 · p. 19

E.5 Reject to Answer (2.6%)

• GPT-4V sometimes refuses or cannot provide an answer, while its calculation errors in this analysis occur only during symbolic calculations, undermining reliability.Source: §E.5 · p. 19, §E.6 · p. 19

F.1 Correct Examples

• GPT-4 with image captions produces correct examples across algebra, arithmetic, analytic geometry, counting, and combinatorics, despite occasional small caption hallucinations.The cited examples are Figures 48–54.Source: §F.1 · p. 19

F.2 Wrong Examples

• Wrong caption-based examples occur on other subjects, where image descriptions are usually inaccurate.The cited examples are Figures 55–60, contrasted with GPT-4V, Gemini Pro, and Qwen-VL-Max in the surrounding comparison.Source: §F.2 · p. 19, F Examples of GPT-4 with Image Captions · p. 19

G Details of Human Annotators

• Human annotators verify question-image alignment, classify problems into 16 subjects, and evaluate human performance on MATH-V.The annotation program uses separate tasks for alignment, subject categorization, and performance measurement.Source: G Details of Human Annotators · p. 19

G.1 Alignment of Text Questions and Images

• Ten senior science-major college students check question-image matching and completeness, recapturing incomplete or mispositioned images for $0.35 per question.This is the first-stage alignment procedure.Source: §G.1 · p. 19

G.2 Subjects Classification

• Subject labels combine independent judgments from 10 annotators and three authors with classifications from Gemini and GPT-4V, using definitions for 16 categories.Annotators are paid $0.45 per question, and classification mistakes are randomly checked and corrected.Source: §G.2 · p. 19

G.3 Human Performance

• Human performance is estimated from 30 college or master students solving two randomly assigned testmini subsets of 30 or 31 questions within 70 minutes across two days.Each participant receives a one-time $45 payment; the time limit is aligned with the Kangaroo competition format.Source: §G.3 · p. 20

Super-CLEVEL

• Detailed image captions enable GPT-4 to solve several arithmetic, algebra, counting, and combinatorics problems that Gemini Pro or Qwen-VL-Max sometimes miss.Examples include correctly identifying cloud D, solving an algebra problem, counting bricks, and solving number-placement or egg-arrangement questions.Source: Figure 48 · p. 59, Super-CLEVEL · p. 60, Figure 49 · p. 60
• In other cases, inaccurate image recognition or descriptions reverse the outcome: GPT-4 fails while GPT-4V succeeds, or GPT-4V misinterprets an image despite giving the correct answer.This occurs in topology, combinatorial geometry, and transformation geometry examples.Source: Figure 55 · p. 66, Figure 56 · p. 67, Figure 57 · p. 68
• Models struggle with spatial relationships in stacked-card diagrams, where the examples report failure to interpret the cards’ order correctly.The task requires recovering the spatial relationships among five overlapping square cards.Source: Figure 58 · p. 69, Super-CLEVEL · p. 69
• GPT-4 fails to find all necessary equations in a sequence-sum problem, whereas GPT-4V correctly solves it.The example attributes GPT-4’s failure to not identifying the necessary equations.Source: Figure 59 · p. 70
• GPT-4 also fails on a metric-geometry problem by missing the correct relationship between variables, while GPT-4V correctly solves it.The problem asks for the radius of a circle inscribed in a quarter circle of radius 6.Source: Figure 60 · p. 71, Super-CLEVEL · p. 71