PaperlayerView exact revision on arXiv

Source-linked AI summary

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li

arXiv:2403.14624v2cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLMs’ visual mathematical reasoning is difficult to evaluate because redundant question text can substitute for diagram interpretation, while final-answer scoring overlooks intermediate reasoning quality. MathVerse addresses these gaps with a multi-version benchmark and GPT-4-based stepwise CoT evaluation, finding that most MLLMs struggle with diagrams and may perform better without them.

  • Problem

    Existing visual math benchmarks provide insufficient evidence that MLLMs genuinely interpret diagrams and inadequately assess their step-by-step reasoning.

  • Method

    MathVerse contains 2,612 multi-subject visual math problems transformed into six versions with varying information content, alongside GPT-4-based extraction and scoring of crucial reasoning steps.

  • Results

    Most MLLMs rely heavily on textual questions and struggle with math diagrams; GPT-4V performs best overall, while Qwen-VL-Max and InternLM-XComposer2 improve without diagrams by 5.1% and 5.6%, respectively.

  • Takeaways & Limitations

    MathVerse provides a benchmark for distinguishing reliance on question text from genuine diagram interpretation and for examining intermediate visual mathematical reasoning.

  • Takeaways & Limitations

    The curated dataset focuses on moderately difficult high-school mathematics, leaving broader complexity and college-level or scientific disciplines for future augmentation.

Abstract

from arXiv · show

The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

1 Introduction

MathVerse identifies benchmark shortcuts and coarse evaluation practices that obscure whether MLLMs genuinely interpret mathematical diagrams, then introduces a specialized benchmark and fine-grained CoT evaluation strategy. Experiments show that most MLLMs rely heavily on textual questions and struggle with diagram understanding.

  • Benchmark Problems: Existing visual math benchmarks often repeat diagram information in question text, allowing MLLMs to solve problems without interpreting the diagrams.Removing redundant text causes accuracy to drop for most MLLMs.
  • Benchmark Problems: Binary final-answer evaluation misses meaningful differences in the quality of models’ intermediate mathematical reasoning.Three models can produce incorrect final answers while exhibiting different reasoning-process quality.
  • Benchmark Problems: Previous benchmarks provide limited coverage of professional mathematical reasoning, either focusing narrowly on plane geometry or including peripheral visual tasks and college-level domain knowledge.These design choices can limit evaluation of functions, solid geometry, or the reasoning capacity of MLLMs.
  • MathVerse: MathVerse collects 2,612 visual math problems across plane geometry, solid geometry, and functions, with expert review and twelve fine-grained categories.The benchmark is designed as a specialized evaluation of multi-modal mathematical reasoning.
  • Findings: Experiments show that most existing MLLMs struggle to understand math diagrams and rely heavily on textual questions, while GPT-4V performs best overall across versions and subjects.Some models improve without diagrams, including Qwen-VL-Max by 5.1% and InternLM-XComposer2 by 5.6%, whereas GPT-4V and ShareGPT4V show relatively better visual comprehension.
  • MathVerse: Each problem is transformed into versions with varying textual and visual information to test whether MLLMs interpret diagrams rather than rely on question text.The benchmark distinguishes descriptive information, implicit properties, and essential conditions according to their problem-solving significance.
  • MathVerse: MathVerse’s CoT evaluation uses GPT-4 to extract crucial solving steps and assess intermediate deductions with detailed error analysis.This provides a finer-grained evaluation of visual mathematical reasoning than judging only the final answer.

2 MATHVERSE

MATHVERSE is a specialized visual math benchmark that varies textual and visual information across six problem versions to evaluate diagram-based mathematical reasoning. It also introduces a CoT evaluation strategy that scores intermediate reasoning and final answers with detailed error analysis.

  • Visual Math Dataset: MATHVERSE contains 2,612 visual math problems spanning plane geometry, solid geometry, and functions, producing 15,672 test instances across six versions.The dataset combines curated problems with systematically transformed multimodal inputs.
  • Visual Math Dataset: Expert review verifies answer accuracy, question–diagram consistency, and relevance to twelve predefined categories.Problems are selected to exclude extremely simple, excessively complex, dubious, or under-specified questions.
  • Data Formulation: The benchmark separates textual content into Descriptive Information, Implicit Property, and Essential Condition according to its visual and problem-solving roles.Descriptive Information repeats visible content, Implicit Property captures visually discernible relations, and Essential Condition provides indispensable numerical or algebraic measurements.
  • Data Formulation: Six versions progressively reduce textual redundancy and move critical information into diagrams, ranging from Text-dominant and Text-lite to Vision-intensive, Vision-dominant, and Vision-only settings.The Text-only version removes the diagram from Text-dominant inputs, while Vision-only requires extracting figures and questions primarily from visual information.
  • CoT Evaluation Strategy: The CoT evaluation strategy scores intermediate reasoning steps and the final answer, using a balancing factor α set to 0.7 to emphasize reasoning.It also supplies detailed step-level error analysis rather than relying only on binary final-answer correctness.

3 Experiments

Experiments evaluate MLLMs on MATHVERSE using the testmini subset and CoT-based analysis. Results show that models often rely on textual information, struggle with diagram-specific mathematical conditions, and exhibit substantial reasoning errors.

  • Experimental setup: 2,612 problems and 15,672 instances comprise MATHVERSE, while testmini contains 788 problems and 4,728 instances used for all quantitative experiments.The subset is randomly sampled across subfields with proportional representation.
  • Visual interpretation: Most MLLMs improve when visual input is removed, including +5.1% for Qwen-VL-Max and +5.6% for InternLM-XComposer2.The result suggests that inaccurate visual encoding of mathematical diagrams can interfere with problem solving.
  • Visual interpretation: Removing Essential Conditions reduces Vision-dominant accuracy by -5.8% for GPT-4V and -3.9% for InternLM-XComposer2.The analysis attributes this difficulty to inaccurate mathematical-symbol identification and insufficient domain knowledge for associating measurements with concepts.
  • Visual interpretation: Vision-only problems further diminish MLLM mathematical problem-solving, challenging models to understand conditions, questions, and figures from images alone.The setting is intended to approximate applications where capturing an image is easier than transcribing the problem.
  • Model comparisons: GPT-4V leads MLLMs, while InternLM-XComposer2 and SPHINX-MoE are the strongest open-source models but trail Gemini-Pro by 9.4% and 12.5% overall accuracy.Closed-source MLLMs consistently outperform open-source models in the reported tables.
  • Model comparisons: GPT-4 outperforms InternLM-XComposer2 and SPHINX-MoE by 12.4% and 18.8% on Text-lite problems despite using only textual questions.The comparison indicates strong mathematical reasoning by text-only LLMs.
  • CoT error analysis: CoT-E exceeds w/o scores by +16.1% for GPT-4V and +9.6% for InternLM-XComposer2, revealing correct intermediate steps despite incorrect final answers.The fine-grained evaluation captures reasoning quality that binary final-answer scoring misses.
  • CoT error analysis: Incorrect final answers are mostly caused by partially incorrect reasoning, while 15.3% of Text Dominant problems have correct answers with partially or entirely incorrect reasoning.Errors involving incorrect reasoning increase as textual content is removed and visual content is emphasized.

4 Conclusion

MATHVERSE is proposed as a specialized benchmark for visual mathematical problem solving. It uses six problem versions and CoT evaluation to assess diagram interpretation and intermediate reasoning, finding that most MLLMs struggle with mathematical diagrams.

  • MATHVERSE collects diagram-based problems across three primary subjects and twelve subfields, transforming each problem into six versions.
  • The benchmark evaluates whether and how much MLLMs interpret visual mathematical diagrams, while its CoT strategy assesses intermediate reasoning more finely.
  • Most existing MLLMs struggle to accurately understand mathematical diagrams and can achieve higher results without visual input.

Appendix Overview

The appendix covers related work, experimental and dataset details, benchmark comparisons, qualitative examples, and limitations or future work.

  • Related Work: Section A reviews related work on MLLMs and mathematical reasoning benchmarks.
  • Additional Materials: Sections B and C provide additional experimental details and dataset details.
  • Additional Materials: Section D compares MATHVERSE with current benchmarks.
  • Additional Materials: Section E discusses limitations and future work, while Section F presents qualitative examples.

B Additional Experimental Details

The appendix documents model sources, prompts, CoT evaluation, human assessment, example function problems, and manual annotations for function diagrams.

  • Experimental details: MATHVERSE evaluates latest models and best-performing configurations across the selected MLLMs.Table 4 records their release times and model sources.
  • Prompting: Response generation uses separate free-form and multiple-choice prompts, with an added reminder for Vision-only questions rendered in images.
  • CoT evaluation: CoT evaluation has two phases: text-only GPT-4 extracts key steps, and GPT-4V performs multi-step scoring using available problem content.Function problems additionally require annotations in the second phase.
  • Human assessment: Ten qualified college students assess human performance, with each student assigned one problem version to prevent cross-version information leakage.
  • Function examples: The appendix includes function examples covering piecewise, quadratic, linear, and paired quadratic functions.
  • Function annotations: Detailed annotations for 534 function problems provide function expressions and properties for diagram-based CoT evaluation.These annotations benefit GPT-4V accuracy for the evaluation.

C.1 Data Curation

MATHVERSE curates visual math problems through expert collection, categorization, review, and manual transformation into multiple problem versions. The curation spans three subjects and supplements diagrams or text to control their information content.

  • Annotation Process: Twelve expert annotators handled data collection, categorization, quality review, problem-version transformation, and function-diagram annotation.The annotators were senior undergraduate and graduate students with strong science backgrounds.
  • Data Collection: Problems were collected from existing datasets and public repositories, with plane-geometry selections emphasizing quality, varied question types, moderate length, diverse diagrams, and suitable difficulty.
  • Problem-Version Transformation: The Text-dominant example retains duplicated descriptive content, while its shorter variant preserves only the essential geometric conditions.
  • Data Categorization and Review: The dataset was categorized into plane geometry, solid geometry, and functions, then divided into twelve finer categories and manually checked for incorrect answers, multiple diagrams, and visual-solution issues.
  • Problem-Version Transformation: Each problem was manually transformed into six versions using annotated diagrams for Vision-dominant problems and rendered text for Vision-only problems.Descriptive Information and Essential Condition were adjusted to control how much textual or visual information remained.

C.2 Subject and Subfield Definition

MATHVERSE organizes visual math problems into three primary subjects and twelve subfields spanning two-dimensional geometry, three-dimensional geometry, and functions. These definitions target diagram understanding alongside mathematical reasoning across varied task types.

  • Overview: MATHVERSE covers plane geometry, solid geometry, and functions, with twelve finer-grained subfields for evaluating diagram understanding and mathematical reasoning.
  • Plane Geometry: Plane geometry studies points, lines, and surfaces in two dimensions and includes five subfields: Length, Area, Angle, Analytic, and Applied Geometry.
  • Plane Geometry: Plane-geometry Length measures distances and perimeters, while Area computes two-dimensional surfaces using formulas and geometric principles.
  • Solid Geometry: Solid geometry examines three-dimensional objects and contains Length, Area, and Volume subfields concerning spatial measurements, surfaces, and enclosed space.Its examples include cubes, cylinders, spheres, and pyramids.
  • Functions: Functions are divided into Function Coordinate, Function Property, Function Expression, and Applied Function problems.These range from reading graph coordinates and properties to analyzing expressions and applying functions in real-world scenarios.

C.3 Detailed Statistics of MATHVERSE

MATHVERSE statistics describe how textual information is removed across problem versions and how the resulting question lengths and dataset composition vary. The version design reduces question text while preserving answer lengths.

  • Dataset Statistics: 534 newly annotated questions correspond to all function problems, while 5,224 newly annotated diagrams support the Vision-dominant and Vision-only versions.Essential Condition or all textual content was integrated with diagrams for these versions.
  • Problem Length Variance: As Descriptive Information, Implicit Property, and Essential Condition are removed, maximum and average question lengths decrease at both word and character levels, while answer lengths remain unchanged.
  • Problem Length Variance: Question-length distributions and their averages decline progressively when Descriptive Information and Essential Condition are omitted from the Text-dominant version.

D Comparison to Current Benchmarks

MATHVERSE compares itself with existing benchmarks across evaluation design, reasoning assessment, task scope, and test-set size. Its design targets diagram interpretation and intermediate reasoning with multiple controlled versions and step-level scoring.

  • Diagram Interpretation: Existing benchmarks often include textual information that repeats diagram content, allowing models to potentially bypass visual understanding; MATHVERSE uses six versions to vary this information.
  • Evaluation Approach: MATHVERSE examines each crucial solution step and provides detailed error analysis, unlike binary True-or-False scoring of final responses.
  • Task Scope: MathVista includes many peripheral tasks, while MMMU problems may require extensive college-level domain knowledge, potentially diverting or hindering specialized mathematical reasoning assessment.
  • Test-Set Size: MATHVERSE provides nearly three times as many test samples as MathVista and twenty times as many as GeoQA+, including six annotated versions of visual math problems.

E Limitation and Future Work

MathVerse identifies scope boundaries and future extensions involving difficulty, educational level, disciplinary breadth, and language coverage. The supplementary examples illustrate transformed problem versions and model-response comparisons.

  • Limitations: Difficulty levels are not yet distinguished, limiting analysis across a spectrum of problem-solving challenges.The authors leave difficulty-based categorization, analogous to MATH’s five levels, for future work.
  • Limitations: The dataset focuses on high-school mathematics with moderate difficulty rather than college-level or scientific problems.The authors propose expanding future versions to broader complexity and disciplines.
  • Limitations: MathVerse and other current mathematical benchmarks are mainly English-language, limiting evaluation of multilingual MLLMs’ linguistic diversity and understanding.The authors propose incorporating multilingual visual math problems.
  • Future Work and Examples: Supplementary materials present six transformed problem versions and compare responses from different MLLMs on Text-lite problems.The examples include GPT-4V, LLaVA-NeXT, and SPHINX-MoE, alongside CoT key-step extraction and scoring.

F.3 Response of Different Problem Versions

The supplementary response comparisons show model solutions across several visual-math problem versions, including prism diagonals, triangle areas, polynomial graphs, function behavior, and applied periodic motion. The examples also expose varied reasoning quality, including correct, partially scored, and option-mismatch responses.

  • Square Prism: 23.89 cm is obtained for the square-prism diagonal DF using two successive Pythagorean-theorem calculations.The solution first computes EF = 15√2 cm, then combines EF with DE = 11 cm to obtain DF ≈ 23.89 cm.
  • Plane Geometry: 60 is the reported area for an isosceles triangle with base 10 and equal sides 13.The reasoning constructs a height of 12 and applies the triangle-area formula.
  • Functions: B, p(x)=(x+1)2(2x+5)(x-3)2, matches a graph whose roots at -1 and 3 have even multiplicity and whose root at -2.5 has odd multiplicity.The response uses intercept behavior to distinguish the candidate polynomial equations.
  • Applied Functions: The highlighted segment in the swinging-chandelier problem is answered as the maximum distance from the chandelier’s hanging point being 2 meters.Another response instead concludes that none of the choices accurately describes the segment, illustrating answer disagreement.
  • CoT Evaluation: The CoT evaluation scores vary across examples, including Multi-step Score 0.56 with Final CoT Evaluation Score 0.69 and perfect scores of 1.Other examples receive scores such as 0.5 and 0.35, or 0.83 and 0.58.
Loading 2403.14624v2…