Source-linked AI summary
Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, Roy Ka-Wei Lee
TL;DR
Existing open-source image-instruction datasets contain few question-answer pairs per image, limiting their use of visual information for multimodal mathematical reasoning. MathV360K combines selected examples from 24 datasets with synthesized questions, and Math-LLaVA fine-tuned on it improves benchmark performance. The paper reports a 19-point gain over LLaVA-1.5 on MathVista’s minitest split, leading performance on Math-V and MathVerse, and validation on MMMU.
Problem
Existing open-source image-instruction datasets provide limited question-answer pairs per image and do not fully exploit visual information for multimodal mathematical reasoning.
Source: §1 · p. 1Method
The authors select 40K image-question-answer samples from 24 datasets, synthesize 320K additional pairs, and fine-tune LLaVA-1.5 with MathV360K.
Source: §1 · pp. 1–2Results
Math-LLaVA improves 19 points over LLaVA-1.5 on MathVista’s minitest split, achieves leading performance on Math-V and MathVerse, and demonstrates generalizability on MMMU.
Source: §6 · p. 9Takeaways & Limitations
The paper concludes that dataset diversity and synthesis are important for enhancing MLLMs’ mathematical reasoning abilities.
Source: §6 · p. 9The dataset contains images, questions, and answers but lacks intermediate reasoning steps and rationales.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive reasoning capabilities, particularly in textual mathematical problem-solving. However, existing open-source image instruction fine-tuning datasets, containing limited question-answer pairs per image, do not fully exploit visual information to enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs (MLLMs). To bridge this gap, we address the lack of high-quality, diverse multimodal mathematical datasets by collecting 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs, creating the MathV360K dataset, which enhances both the breadth and depth of multimodal mathematical questions. We introduce Math-LLaVA, a LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach significantly improves the multimodal mathematical reasoning capabilities of LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V on MathVista's minitest split, and yielding leading performance on Math-V and MathVerse. Furthermore, Math-LLaVA demonstrates enhanced generalizability, showing substantial improvements on the MMMU benchmark. Our research highlights the importance of dataset diversity and synthesis in advancing MLLMs' mathematical reasoning abilities. The code and data are available at: \url{https://github.com/HZQ950419/Math-LLaVA}.
1 Introduction
Open-source MLLMs struggle with multimodal mathematical problems, and existing image-instruction datasets provide too few question-answer pairs per image to fully use visual information. The paper builds a larger, more diverse dataset and fine-tunes LLaVA-1.5, reporting stronger benchmark performance.
Motivation
- Existing open-source MLLMs fall short of proprietary models on complex mathematical problems involving visual content.The task requires interpreting diverse images and applying advanced reasoning skills.Source: §1 · p. 1
- Existing open-source image-instruction datasets contain limited question-answer pairs per image, leaving visual information underused for multimodal mathematical reasoning.The paper identifies prompting and fine-tuning as common approaches to improving these skills.Source: §1 · p. 1
Research Objectives
- MathV360K combines 40K selected images and question-answer pairs from 24 datasets with 320K synthesized pairs spanning multiple subjects and reasoning types.Selection considers image clarity and comprehension complexity; synthesis uses the selected images and seed inquiries.Source: §1 · pp. 1–2
Contributions
- Math-LLaVA outperforms LLaVA-1.5 by 19% on MathVista’s minitest split and achieves leading performance on Math-V and MathVerse.The paper also reports improved generalizability on MMMU.Source: §1 · p. 2
2 Related Works
Prior multimodal models use visual-language alignment and instruction tuning, but mathematical reasoning with diagrams remains challenging. Existing text augmentation methods do not fully use raw-image information.
2.1 Multimodal Large Language Models
- Models including MiniGPT-4, LLaVA, and InstructBLIP use visual-feature alignment approaches, yet MLLMs still struggle with diagram-based multimodal mathematics.The cited approaches use learnable projectors or query embeddings, among other methods, to interact with visual features.Source: §2.1 · p. 2
2.2 Multimodal Reasoning
- Prior question and answer augmentation methods target text within restricted domains, neglecting visual information in raw images.The section also identifies MathVista, Math-V, MathVerse, and MMMU as benchmarks for evaluating multimodal reasoning.Source: §2.2 · p. 2
3 Data Synthesis
MathV360K is built by selecting 40K image-question-answer samples from existing datasets, then expanding and varying their questions. The pipeline filters for image quality and complexity and uses clustered examples to guide question generation.
3 Data Synthesis
- MathV360K addresses the limited per-image question coverage in existing datasets by selecting 40K high-quality data points and generating additional questions.The selection and augmentation pipeline yields 360K instruction-tuning samples for the selected images.Source: §3 · pp. 2–3
3.1.1 Source Data
- The 24 source datasets cover five task types: figure question answering, geometry problem solving, math word problems, textbook question answering, and visual question answering.Each sample consists of an image, a text question, and a ground-truth answer.Source: §3.1.1 · p. 3
3.1.2 Image Filtering and Proportioning
- Image selection filters for clarity, balances comprehension complexity, and checks that question difficulty aligns with image complexity.GPT-4V labels 10K sampled images for clarity and complexity, and two fine-tuned ViT models classify those properties across the source images.Source: §3.1.2 · pp. 3–4
3.2.1 Mining Image for QA Generation
- For 200K new pairs, GPT-4V uses few-shot demonstrations sampled from question clusters to generate task-appropriate, distribution-aligned questions.Questions are clustered by TF-IDF features and K-Means, with each cluster representing a questioning format or pattern.Source: §3.2.1 · p. 4
3.2.2 Augmentation of Original Question
- Question augmentation adds 40K complex, 40K simplified, and 40K rephrased questions to vary reasoning demands and phrasing.The rephrased questions preserve answers, while simplified questions are intended to remain understandable when combined with the image.Source: §3.2.2 · pp. 4–5
4 Experiments
The experiments fine-tune LLaVA-1.5-13B on MathV360K and evaluate it on mathematical and multidisciplinary benchmarks. The supplied passages describe the model, evaluation setup, and training configuration.
4.1 Model and Training
- Math-LLaVA uses the LLaVA-1.5 architecture with Vicuna-v1.5 and a pretrained ViT image encoder, fine-tuned on MathV360K.The stated aim is to improve multimodal mathematical reasoning while maintaining general vision-language understanding.Source: §4.1 · p. 5
4.2 Evaluation and Metrics
- Evaluation includes zero-shot MathVista minitest accuracy, Math-V, MathVerse, and MMMU, which covers six core disciplines.MathVista minitest has 1,000 samples; MMMU has 900 evaluation samples.Source: §4.2 · pp. 5–6
4.3 Implementation Details
- Training uses 336-by-336 images and fine-tunes the projection layer and language model for two epochs with batch size 16.The reported learning rate is 2e-5, using A800 GPUs with 80GB memory.Source: §4.3 · p. 6
5 Results and Analysis
Math-LLaVA is compared with other models on MathVista, then evaluated for generalization and for possible text-only overfitting. Additional experiments test the effects of data selection and augmentation, both overall and by task type.
5.1 Main Comparison on MathVista
- 46.6% overall MathVista accuracy and a 19% improvement over LLaVA-1.5-13B put Math-LLaVA near GPT-4V and above Gemini 1.0 Pro and Claude 3 Haiku.It also reaches 57.7% accuracy on the GPS subset, outperforming G-LLaVA-13B; the prose reports leading performance on Math-V and improvement on MathVerse.Source: §5.1 · p. 6, Table 1 · p. 7
5.2 Generalizability of Math-LLaVA
- On MMMU, Math-LLaVA fine-tuned with MathV360K outperforms LLaVA-1.5-13B and several open-source MLLMs across all six subdomains, supporting generalization to multimodal understanding and reasoning.The selected-data-only model drops on the science subset; the MathV360K model improves across all six subdomains.Source: §5.2 · p. 7, Table 3 · p. 8
5.3 Overfitting to Text Modality
- About 32.0% text-only MathVista performance was similar for Math-LLaVA and Math-LLaVA-DS; LLaVA-1.5-13B also scored 23.3% without images, which the authors note may reflect image-unnecessary questions or pretraining leakage.The authors interpret the similar scores before and after synthesis as evidence that Math-LLaVA is not overfitting to synthesized pairs.Source: §5.3 · pp. 7–8, Table 4 · p. 8
5.4 Effectiveness of Synthesis
- 11% improvement over random 40K sampling followed the combined use of data selection and four augmentation strategies on MathVista.The strategies combine selected data with AskImg, CompQ, RephQ, and SimpQ.Source: §5.4 · p. 8, Table 5 · p. 9
5.5 Enhancements from Augmentation of Each Task Type
- Augmenting individual task types particularly improved FQA, MWP, and VQA, and sometimes benefited other task types, possibly through overlapping reasoning skills.Figure 6 reports MathVista accuracy by task type and augmentation.Source: Figure 6 · p. 9, §5.5 · p. 9
6 Conclusions
The paper introduces MathV360K and uses it to fine-tune Math-LLaVA. The authors report improved benchmark performance and generalizability, while identifying the absence of intermediate reasoning steps in the data.
6 Conclusions
- MathV360K combines 40K image question-answer pairs from 24 datasets with 320K synthesized pairs, and Math-LLaVA improves on LLaVA-1.5 by 19 points on MathVista minitest.The paper also reports leading performance on Math-V and MathVerse and validation on MMMU.Source: §6 · p. 9
- The selected and synthesized data contain images, questions, and answers but lack intermediate reasoning steps.The authors propose adding annotated intermediate steps and rationales in future work.Source: §6 · p. 9
- The authors conclude that dataset diversity and synthesis are important for advancing MLLM mathematical reasoning within the evaluated scope.The reported evidence includes MathVista, Math-V, MathVerse, and MMMU.Source: §6 · p. 9
7 Limitations
The selected and synthesized training data provide image-question-answer examples without intermediate steps.
7 Limitations
- The data lack intermediate reasoning steps, which the authors identify as an area for improving dataset comprehensiveness and quality.They propose adding annotated steps and rationales in future work.Source: §7 · p. 9
8 Ethics Statement
The authors state that they do not anticipate harm as defined by their ethics policy and describe the intended use and licensing of the model and source datasets.
8 Ethics Statement
- The authors do not envision that the work will result in harm as defined by their ethics policy.Source: §8 · p. 9
- The stated intended use of the source and evaluation datasets is training and testing multimodal reasoning capability.Source: §8 · p. 9
- The authors report that the data and model are publicly available and list licenses or access terms for the base model and datasets.The listed terms include Apache, MIT, BSD, GPL, research-only availability, and permissive Creative Commons licenses.Source: §8 · p. 9
A.1 Source Data Statistics
The source data draw on 24 visual question-answering and multimodal mathematical reasoning datasets spanning five task types and varied visual contexts.
A.1 Source Data Statistics
- The source collection combines 24 datasets covering visual question answering and multimodal mathematical reasoning.The datasets target different task types and visual content.Source: §A.1 · p. 13
- Five task types organize the data: figure question answering, geometry problem solving, math word problems, textbook question answering, and visual question answering.They respectively cover charts and plots, geometry diagrams, arithmetic in image contexts, scientific questions with figures, and objects, scenes, or relationships.Source: §A.1 · p. 13
- Combining these domains supports coverage of multiple tasks, visual contexts, and mathematical skills.Textbook and visual question answering also include some questions requiring arithmetic or numeric skills.Source: §A.1 · p. 13
A.2 Results on MathVerse Benchmark
Math-LLaVA improves over its base model on MathVerse and shows stronger performance in the reported complexity and case studies. The examples also illustrate diverse question generation and improved handling of underspecified questions.
A.2 Results on MathVerse Benchmark
- Math-LLaVA significantly improves over the base model and performs impressively among most open-source MLLMs on MathVerse.The comparison reports accuracy; Table 7 presents MathVerse benchmark results.Source: §A.2 · p. 13, Table 7 · p. 14
A.3 Distribution Proportioning of Image Comprehension Complexity
- Progressive sampling aligned with image complexity outperforms uniform and decreasing-proportion distributions on MathVista.The authors conclude that training benefits from fewer simple images and a larger share of complex data.Source: §A.3 · pp. 13, 15, Table 8 · p. 15
A.4 Cases Study
- In high-school problems involving tables, geometry, and counting, Math-LLaVA outperforms LLaVA-1.5, while showing promise on advanced functions and detailed tables.Figure 7 presents college- and high-school-level cases.Source: §A.4 · p. 15, Figure 7 · p. 16
- Few-shot question generation mines images from varied perspectives, adding more complex, rephrased, and simplified underspecified questions to the dataset.The generated questions follow existing question formats and include image-based reasoning.Source: §A.4 · p. 15, Figure 8 · p. 17
- Math-LLaVA answers underspecified and logically consistent rephrased questions more robustly than LLaVA-1.5 in the reported examples.Figure 9 illustrates the comparisons.Source: §A.4 · p. 15, Figure 9 · p. 18
Grade Level: College
- On the equilibrium-population example, Math-LLaVA selects 80 correctly while LLaVA-1.5 selects 40; other examples show gains on underspecified questions.Math-LLaVA also answers the market-situation and smallest-element questions correctly where LLaVA-1.5 does not.Source: Grade Level: College · pp. 16, 18