Source-linked AI summary
MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts
Peijie Wang, Zhong-Zhi Li, Fei Yin, Xin Yang, Dekang Ran, Cheng-Lin Liu
TL;DR
Existing multimodal mathematics benchmarks largely focus on single-visual contexts, leaving multi-visual reasoning insufficiently assessed. The paper introduces MV-MATH, a curated multi-image benchmark built from authentic K-12 problems, and finds substantial challenges for current MLLMs relative to humans. The study also analyzes model performance across visual dependencies, input methods, question types, and prompting conditions.
Problem
Most multimodal math benchmarks use single-visual contexts, while multi-visual mathematical reasoning remains insufficiently evaluated.
Method
MV-MATH provides 2,009 cross-validated, interleaved multi-image mathematics problems from authentic K-12 scenarios, spanning question types, subjects, difficulty levels, and image-relevance categories.
Results
Claude-3.5-Sonnet achieves 33.9% accuracy versus 76.5% for humans, while performance varies with image relevance, input method, question type, and prompting.
Takeaways & Limitations
MV-MATH establishes a challenging benchmark showing that current MLLMs still have substantial room to improve multi-visual mathematical reasoning and generalization.
Takeaways & Limitations
Math-LLaVA trained on Math360K does not effectively generalize to MV-MATH, highlighting a limitation in generalization capabilities.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown promising capabilities in mathematical reasoning within visual contexts across various datasets. However, most existing multimodal math benchmarks are limited to single-visual contexts, which diverges from the multi-visual scenarios commonly encountered in real-world mathematical applications. To address this gap, we introduce MV-MATH: a meticulously curated dataset of 2,009 high-quality mathematical problems. Each problem integrates multiple images interleaved with text, derived from authentic K-12 scenarios, and enriched with detailed annotations. MV-MATH includes multiple-choice, free-form, and multi-step questions, covering 11 subject areas across 3 difficulty levels, and serves as a comprehensive and rigorous benchmark for assessing MLLMs' mathematical reasoning in multi-visual contexts. Through extensive experimentation, we observe that MLLMs encounter substantial challenges in multi-visual math tasks, with a considerable performance gap relative to human capabilities on MV-MATH. Furthermore, we analyze the performance and error patterns of various models, providing insights into MLLMs' mathematical reasoning capabilities within multi-visual settings.
1. Introduction
Existing multimodal math benchmarks largely use single images, leaving multi-visual mathematical reasoning insufficiently evaluated. MV-MATH addresses this gap with a curated, annotated benchmark and experiments showing substantial model–human performance differences and varied model behaviors.
- Most multimodal mathematics datasets require reasoning from a single image, unlike the multi-visual scenarios targeted by MV-MATH.
- MV-MATH contains 2,009 authentic K-12 mathematics problems with interleaved multi-visual contexts and cross-validated questions, answers, and images.
- The dataset includes 1,109 multiple-choice, 900 free-form, and 100 multi-step questions across three difficulty levels and 11 mathematical subjects.
- Claude achieves 33.9% on MV-MATH, while LLaVA-OneVision reaches 26.2% and remains below human-level capability.
- Analysis finds weaker performance on image-dependent tasks, better results for sequential than merged image input, and inconsistent gains from chain-of-thought prompting.
2. Related Work
Prior mathematical benchmarks span pure-text and multimodal reasoning, but most multimodal datasets remain single-visual. MV-MATH instead emphasizes relationships among multiple images to better represent multi-visual mathematical reasoning.
- Pure Text Benchmarks: Pure-text benchmarks evaluate mathematical reasoning through natural-language problems ranging from grade-school arithmetic to advanced multi-step tasks.
- Multimodal Benchmarks: Multimodal benchmarks such as MathVista, MATH-Vision, and MathVerse evaluate mathematical reasoning with visual elements.
- Limitations of Existing Benchmarks: Existing multimodal datasets generally use one image per problem, limiting the need for information interaction between visual elements.
- MV-MATH: MV-MATH incorporates multiple images per problem, requiring models to understand relationships among mathematical visual elements.
- Multimodal Models: Vision-language alignment and instruction tuning have produced models including CLIP, BLIP, LLaVA, GPT-4V, Gemini, Claude, and LLaVA-OneVision.
3. The MV-MATH Benchmark
MV-MATH is an interleaved multi-image mathematics benchmark designed for fine-grained evaluation across question types, subjects, difficulty levels, and image-relevance conditions. Its construction combines large-scale collection, multi-stage filtering, manual verification, and comparative benchmark analysis.
- Overview: Each MV-MATH sample interleaves multiple images with text, imposing greater demands on multimodal fusion and comprehension.
- Overview: The benchmark contains 2,009 questions, up to eight images per question, three question types, 11 subjects, and three difficulty levels.
- Overview: Images are categorized as mutually dependent when one requires information from another, or independent when images can be interpreted separately.
- Data Construction: Collection began with over 300,000 mathematics problems and 49,538 preliminary multi-image problems before staged filtering and quality screening.
- Data Construction: The final screened set contains 1,109 multiple-choice, 800 one-step free-form, and 100 multi-step free-form questions, with each step cross-verified by at least two graduate students.
- Comparison with Existing Benchmarks: Compared with MathVerse-mv, MV-MATH offers more authentic K-12 samples, broader question formats, and richer difficulty and subject coverage.
- Comparison with Existing Benchmarks: The coefficient of variation is defined as the standard deviation of sample lengths divided by their mean, multiplied by 100%.
- Comparison with Existing Benchmarks: Compared with CMM-Math, MV-MATH emphasizes manually checked image quality and primarily English-language scenarios.
4. Experiments
Experiments evaluate diverse models and prompting conditions on MV-MATH, revealing low accuracy, modality-dependent difficulty, and sensitivity to image organization and question type. Performance is especially weak for mutually dependent images, complex visual subjects, and multi-step free-form reasoning.
- Sequential image input outperforms merged input, while mutually dependent image tasks are more challenging than independent-image tasks.
- Claude-3.5-Sonnet reaches 33.9% accuracy, versus 76.5% for humans, establishing a substantial performance gap on MV-MATH.
- Math-LLaVA achieves 3.0% accuracy and does not effectively generalize from Math360K to MV-MATH.
- Claude achieves 27.0% on combinatorial geometry but 54.2% on arithmetic, indicating weaker performance in subjects requiring complex image understanding.
- CoT and few-shot prompting do not consistently improve performance, and adding 2-shot examples tends to degrade results for open-source models.
- GPT-4o reaches only 6% QCR and 32.6% SAR on multi-step questions, while multiple-choice accuracy exceeds free-form accuracy.
5. Error Analysis and Model Insights
Error analysis of Claude-3.5-Sonnet identifies visual perception as the dominant failure category, while model-insight analyses connect multi-image training and preference optimization with stronger LLaVA-OV variants.
- Error Analysis: The error analysis examines 100 randomly sampled Claude-3.5-Sonnet errors and classifies them into five types.
- Error Analysis: Visual perception errors account for 42% of sampled Claude errors, exceeding proportions reported for most single-image datasets.
- Error Analysis: Multi-visual perception requires both accurate interpretation of individual images and understanding of relationships between images.
- Model Insights: LLaVA-OV-Chat builds on multi-image training with Direct Preference Optimization and human feedback to improve generalization and reasoning in multi-visual contexts.
6. Conclusion
MV-MATH addresses the evaluation gap for multi-image mathematical reasoning with a diverse benchmark and systematic model analysis. Its results expose substantial human–MLLM performance differences and identify image relevance, input methods, and question types as important factors.
- MV-MATH provides a diverse benchmark for evaluating MLLMs’ mathematical reasoning in multi-visual contexts.
- The benchmark reveals a substantial performance gap between human capabilities and current MLLMs on multi-visual mathematical reasoning.
- The study highlights the impact of image relevance, image input methods, and question types on multi-visual mathematical performance.
Supplementary Material
MV-MATH supplements detailed dataset statistics with a two-step difficulty classification and broad coverage across mathematical subjects and visual styles.
- Dataset Statistics: MV-MATH questions average 80.17 words, range from 14 to 383 words, and average 3.02 images with a median of 3.Most questions contain 2, 3, or 4 images, while some contain 5 or more.
- Difficulty Classification: Difficulty classification combines weighted question and analysis lengths, K-means clustering, and manual verification by two graduate students.Question length receives weight 0.4 and analysis length 0.6; manual review can revise classifications for unusually complex or straightforward problems.
- Subject Coverage: The dataset spans 11 mathematical categories, including Algebra, Arithmetic, Geometry, Combinatorics, Logic, Statistics, and Transformations.Its images range from charts, graphs, and tables to playful real-world designs, diagrams, coordinate systems, vectors, and precise measurements.
- Subject Coverage: MV-MATH’s varied visual styles support evaluation of mathematical interpretation and reasoning across diverse visual contexts.The supplementary figures provide examples from each subject category.
B. Evaluation Details
The evaluation uses image captioning, task-specific prompting, answer extraction, and controlled generation settings across multiple question formats.
- Evaluation Setup: Claude-3.5-Sonnet generates detailed textual descriptions of mathematical diagrams for image-caption-based evaluation.The captions are intended to replace the diagrams with textual descriptions.
- Evaluation Setup: Distinct original, Chain-of-Thought, and 2-shot prompts are designed for multiple-choice, single-step, and multi-step free-form questions.The prompts are tailored both to question type and to response generation.
- Evaluation Setup: The Deepseek API extracts model answers and compares them with ground truth for correctness assessment.Different evaluation prompts are used for multiple-choice, single-step, and multi-step responses.
- Evaluation Setup: All models use temperature 0.2 and a maximum of 2048 newly generated tokens.Model sources and configurations are documented separately.
C. Main Results Across 3 Question Types
Across multiple-choice, single-step free-form, and multi-step free-form questions, model performance declines as response formats demand more open-ended and complete reasoning.
- Multiple-Choice Questions: Claude-3.5-Sonnet leads multiple-choice questions with 44.0% accuracy.GPT-4o and LLaVA-OneVision-Chat-72B follow it on this question type.
- Single-Step Free-Form Questions: Claude-3.5-Sonnet also leads single-step free-form questions with 24.1% accuracy.The performance gap between open-source and closed-source models widens on this format.
- Multi-Step Free-Form Questions: GPT-4o achieves the highest multi-step scores, with 32.0% Step Accuracy Rate and 6.0% Question Completeness Rate.The overall multi-step accuracy reported in Table 4 uses QCR.
- Multi-Step Free-Form Questions: The 6.0% Question Completeness Rate indicates that models struggle to complete complex multi-visual mathematical reasoning tasks.The study evaluates the three question types separately to compare complexity and reasoning requirements.
D. Results of CoT, 2-shot on 3 Question Types
Prompting effects depend on question type: simpler prompts generally work best for multiple-choice questions, whereas Chain-of-Thought and 2-shot prompting help free-form reasoning, especially on multi-step tasks.
- Multiple-Choice Questions: Eight of ten tested models perform best with the original prompt on multiple-choice questions.Adding Chain-of-Thought and 2-shot examples tends to decrease performance for this format.
- Free-Form Questions: Claude-3.5-Sonnet improves from 19.6% to 25.6% on single-step free-form questions with Chain-of-Thought and 2-shot prompting.The strategies produce larger gains on free-form formats than on multiple-choice questions.
- Free-Form Questions: GPT-4o improves from 25.4% to 32.6% on multi-step free-form questions when Chain-of-Thought and 2-shot strategies are added.The improvement is reported for questions requiring detailed reasoning steps.
- Cross-Format Findings: Prompting strategies do not uniformly improve performance, but they are particularly effective for free-form questions requiring detailed reasoning.The study also examines image relevance and finds that relevance matters more than image count for multi-image reasoning.
F. Data Collection and Annotation
MV-MATH uses staged filtering and annotation procedures to ensure that retained questions and images are aligned, complete, and suitable for multimodal mathematical evaluation.
- Data Collection: A three-stage process filters questions for text-image alignment, missing fields, and image quality.The stages combine rule-based checks with manual verification.
- Data Collection: Text-image alignment requires the number of referenced images to match the images returned by the Mathpix API.Mismatched image counts trigger filtering by the rule-based system.
- Data Collection: Rule-based filtering detects missing text fields and separates samples into multiple-choice and free-form subsets.The missing-field checks include absent answers or analyses.
- Annotation: Manual verification removes blurred images and images with text overlays.The annotation team includes graduate students and a field expert.
- Annotation: Subject and image-relevance labels begin with majority voting by GPT-4o, Claude-3.5-Sonnet, and Qwen-VL-Max, followed by human review and expert adjudication.Questions without model consensus receive no initial annotations; conflicts between graduate-student annotations are resolved by domain experts.
G. Comparison with Existing Benchmarks
The paper compares MV-MATH with existing multimodal mathematics benchmarks and documents model evaluations across question types, subjects, and categorized error patterns.
- Error Analysis: The paper categorizes Claude-3.5-Sonnet’s errors into visual perception, reasoning, calculation, knowledge, and reject errors.Figures 21–30 provide detailed examples of these five error categories.
- Benchmark Comparison: Tables 10 and 11 compare model performances across mathematical subjects for choice and single-step problem sets.The tables mark the first and second highest model accuracies.
- Benchmark Comparison: Tables 12 and 13 compare model performances across mathematical subjects on SAR and QCR multi-step problem sets.Table 12 is identified as the SAR multi-step problem set, while Table 13 is identified as the QCR multi-step problem set.
- Benchmark Comparison: Table 14 evaluates model performance across three question types and configurations.It marks the best performance in each category and indicates configuration changes with red or green annotations.
- Benchmark Comparison: Table 15 compares MV-MATH with existing multimodal mathematics benchmarks across MC, FF, and MS question types.MC denotes multiple choice, FF free-form, and MS multi-step.