PaperlayerView exact revision on arXiv

Source-linked AI summary

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, Hongsheng Li

arXiv:2505.10557v1cs.CVcs.AIcs.CL

TL;DR

LMMs lack reliable math-specific cross-modal data because natural-image captions omit important figure details. MathCoder-VL uses code-supervised image alignment and synthesized multimodal instruction data, achieving state-of-the-art performance among open-source models, including on MathVista GPS against GPT-4o and Claude3.5-Sonnet.

  • Problem

    Natural-language image-caption datasets often omit mathematical figure details and cannot guarantee correctness for multimodal math reasoning.

  • Method

    The paper co-develops FigCodifier and ImgCode-8.6M for code-supervised alignment, then uses synthesized figures to construct MM-MathInstruct-3M for instruction fine-tuning.

  • Results

    MathCoder-VL achieves a new state-of-the-art among open-source multimodal mathematical-reasoning models and outperforms GPT-4o and Claude3.5-Sonnet on MathVista GPS.

  • Takeaways & Limitations

    The model-based data engine supports accurate cross-modal alignment and multimodal math problem-solving data with diverse newly synthesized images.

  • Takeaways & Limitations

    MM-MathInstruct-3M focuses on mathematics, uses English text only, and the study trains only 2B and 8B models.

Abstract

from arXiv · show

Natural language image-caption datasets, widely used for training Large Multimodal Models, mainly focus on natural scenarios and overlook the intricate details of mathematical figures that are critical for problem-solving, hindering the advancement of current LMMs in multimodal mathematical reasoning. To this end, we propose leveraging code as supervision for cross-modal alignment, since code inherently encodes all information needed to generate corresponding figures, establishing a precise connection between the two modalities. Specifically, we co-develop our image-to-code model and dataset with model-in-the-loop approach, resulting in an image-to-code model, FigCodifier and ImgCode-8.6M dataset, the largest image-code dataset to date. Furthermore, we utilize FigCodifier to synthesize novel mathematical figures and then construct MM-MathInstruct-3M, a high-quality multimodal math instruction fine-tuning dataset. Finally, we present MathCoder-VL, trained with ImgCode-8.6M for cross-modal alignment and subsequently fine-tuned on MM-MathInstruct-3M for multimodal math problem solving. Our model achieves a new open-source SOTA across all six metrics. Notably, it surpasses GPT-4o and Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista, achieving improvements of 8.9% and 9.2%. The dataset and models will be released at https://github.com/mathllm/MathCoder.

1 Introduction

LMMs struggle with math-specific visual-textual alignment because existing caption data often omits important figure details and cannot guarantee correctness. MathCoder-VL addresses these challenges with code-supervised alignment and synthesized multimodal math data.

  • LMMs still struggle with multimodal mathematical reasoning, including simple geometry problems.
  • The paper identifies accurate alignment of math-related visual and textual details and scalable generation of diverse math figures as two key challenges.
  • Traditional image-caption datasets focus on natural scenarios, lose details important for math problem-solving, and cannot guarantee correctness.
  • FigCodifier converts math images into rendering code, enabling accurate ImageC-Code pairs that contain the figure details needed for cross-modal alignment.
  • FigCodifier can synthesize more diverse images, supporting construction of MM-MathInstruct-3M with new questions and newly synthesized figures.
  • The model-based pipeline produces FigCodifier and ImgCode-8.6M, described as the largest image-code dataset to date.
  • MathCoder-VL achieves state-of-the-art results across all six metrics among comparable-size LMMs.

2 Related Works

Related work spans multimodal mathematical reasoning and synthetic-data methods. Existing multimodal synthesis efforts have emphasized problem or solution text more than diverse mathematical imagery.

  • Multimodal Math Reasoning: Multimodal mathematical reasoning requires extracting information from images and performing cross-modal reasoning, unlike traditional text-only mathematical reasoning.
  • Data Synthesis: Synthetic-data research has explored fine-tuning and pretraining with language-model- or template-generated data.
  • Data Synthesis: Figure 2 situates MathCoder-VL’s approach as an image-to-code pipeline that scales image-code pairs and generates new multimodal math problems.
  • Data Synthesis: Existing multimodal math synthesis methods primarily vary problem or solution text, while MathV360K enhances questions by classifying images by complexity.

3 MathCoder-VL

MathCoder-VL is built through image-to-code mid-training followed by multimodal math instruction fine-tuning. Its data engine iteratively creates validated code-image pairs and generates new questions and solutions from synthesized mathematical figures.

  • Training Process: MathCoder-VL uses image-to-code mid-training on ImgCode-8.6M followed by math instruction fine-tuning on MM-MathInstruct-3M.
  • Image-to-Code Data: The image-to-code effort addresses weak existing image-to-code models and the limited 119K-pair DaTikZ dataset.
  • Image-to-Code Data: The data collection begins with 3 million math-related images, including K12 problems spanning 19 subjects and 1.57 million images.
  • Image-to-Code Data: An initial image-to-code model is trained on DaTikZ pairs and later scaled with InternVL2-8B as the base model.
  • Image-to-Code Data: Iterative synthesis translates collected images into code, renders them, retains successful ImageC-Code pairs, and refines the model with new versions.
  • Image-to-Code Data: TikZ-to-Python conversion broadens code diversity and produces 3.1 million image-Python pairs.
  • Image-to-Code Data: Cleaning validates renderable code, removes 4.4% of the dataset as duplicates, and retains 4.3M image-TikZ and 4.3M image-Python pairs.
  • Multimodal Instruction Data: MM-MathInstruct-3M starts from 4.6 million math problems, converts equation images to text, and expands simple solutions into step-by-step solutions.

4 Experiments

MathCoder-VL is evaluated through two-stage training, broad mathematical benchmarks, comparisons with open- and closed-source models, geometry analysis, and ablations of its training components. It shows strong benchmark performance, especially on geometry and multimodal settings with greater visual dominance.

  • Training Stages: The two-stage pipeline uses ImgCode-8.6M for image-to-code mid-training, then MM-MathInstruct-3M for multimodal math instruction fine-tuning.The first stage trains the vision encoder and MLP projector to improve mathematical cross-modal alignment; the second fine-tunes the entire model.
  • Experimental Setup: The evaluation covers MATH-Vision, MathVerse, MathVista GPS, GAOKAO-MM, and We-Math across English and Chinese mathematical challenges.MathVista results are reported on the Geometry Problem Solving subset because many other tasks emphasize natural-image recognition over mathematical reasoning.
  • Main Results: 26.1% on MATH-Vision, 46.5% on MathVerse, and 73.6% on MathVista GPS make MathCoder-VL-8B the strongest comparable-size open-source LMM on these benchmarks.It improves over InternVL2-8B by 6.1%, 10.6%, and 11.6% on the respective benchmarks.
  • Main Results: MathCoder-VL-8B exceeds GPT-4o and Claude3.5-Sonnet on MathVista GPS, reaching 73.6% versus 64.7% and 64.4%, respectively.It remains behind GPT-4o by 3.0% on MATH-Vision, showing that performance varies across benchmarks.
  • Geometry Analysis: MathCoder-VL-8B scores 37.6% across MATH-Vision plane-geometry subsets, surpassing GPT-4o by 11.9%.Its angle, area, and length subset scores are 48.6%, 32.2%, and 32.1%, respectively.
  • Ablation Study: Image-to-code mid-training improves MATH-Vision by 1.7%, MathVerse by 5.8%, MathVista GPS by 18.7%, and GAOKAO-MM Math by 3.8%.Across modality-dominance settings, the largest gains occur for Vision-Only inputs, at 11.0% and 7.4%.
  • Ablation Study: Newly synthesized images improve MathVerse by 2.4%, MathVista-Testmini by 5.0%, MathVista-GPS by 2.0%, and GAOKAO-MM Math by 3.7%, while reducing MATH-Vision by 0.3%.The reported MathVista increase is 5.0%, whereas the MATH-Vision change is a slight decrease.

5 Conclusion

The paper introduces a model-based multimodal data engine that produces datasets for cross-modal alignment and multimodal mathematical problem solving, enabling MathCoder-VL to achieve open-source state-of-the-art performance.

  • 5 Conclusion: The model-based data engine constructs ImgCode-8.6M for accurate cross-modal alignment and MM-MathInstruct-3M with diverse newly synthesized mathematical images.These datasets support distinct stages of multimodal mathematical reasoning development.
  • 5 Conclusion: MathCoder-VL-2B and 8B use image-to-code mid-training followed by math instruction fine-tuning.
  • 5 Conclusion: MathCoder-VL achieves new state-of-the-art performance among open-source models for multimodal mathematical reasoning.

6 Limitations

The work is scoped primarily to English mathematics and 2B and 8B models, without reinforcement-learning post-training.

  • 6 Limitations: MM-MathInstruct-3M focuses primarily on mathematics and excludes other STEM subjects such as physics and chemistry.
  • 6 Limitations: The dataset contains only English text and does not include math-related content in other languages such as Chinese.
  • 6 Limitations: Computational constraints limited training to 2B and 8B models.
  • 6 Limitations: The paper does not apply post-training reinforcement learning methods such as GRPO.

A.1 Data Cleaning

The data-cleaning process converts equation-only images into LaTeX equations so the curated dataset contains multimodal math problems with consistent textual and visual representations.

  • A.1 Data Cleaning: The cleaning process targets multimodal math problems containing textual descriptions, LaTeX mathematical expressions, and math figures.
  • A.1 Data Cleaning: MinerU converts equations supplied solely as images into LaTeX-formatted equations.This addresses raw examples in which equations appear only as images.

A.2 Data Augmentation

The augmentation pipeline generates mathematical images through TikZ and Python code, using structured prompts and code translation to preserve visual fidelity across representations.

  • A.2 Data Augmentation: The pipeline processes, solves, and formats K-12 math problems from structured JSON input.The workflow includes translation, step-by-step solution generation, and concise answer presentation.
  • A.2 Data Augmentation: TikZ provides programmable, customizable vector graphics integrated with LaTeX documents.The paper illustrates TikZ-generated images as one source of augmented mathematical figures.
  • A.2 Data Augmentation: Matplotlib provides flexible Python-based plotting and visualization capabilities for generating customized figures.The paper includes examples of images generated with Python.
  • A.2 Data Augmentation: Matplotlib uses a procedural syntax, whereas TikZ uses a declarative style with greater control over visual details.The comparison concerns how the two systems specify plots and graphics.
  • A.2 Data Augmentation: Structured prompt templates guide conversion of visual elements into executable code.A GPT4o-mini prompt translates TikZ code into executable Python that reproduces visual details and avoids element overlap.
  • A.2 Data Augmentation: Python-generated images are highly similar to images generated from the original TikZ code.This comparison supports the effectiveness of the TikZ-to-Python translation approach.

B.4 Data Cleaning

The data-cleaning process removes malformed, low-information, and externally dependent image-code pairs while using prompts to generate code representations in TikZ and Python Matplotlib.

  • Images with pixel-value standard deviation below five are removed as almost blank samples.
  • Random lines or shapes are filtered by analyzing and filtering the corresponding code.
  • Image-to-code prompts request precise, faithful TikZ or Python code that reproduces the visual elements in an image.
  • Incorrect preprocessing can create completely black images, which are removed after conversion logic is optimized.
  • Externally loaded content is removed by detecting code commands that access local files.

B.5 Performance of img2code model

The img2code model improves through iterative training on increasingly large image-code datasets and supports high-temperature synthesis of diverse mathematical images and reasoning problems.

  • The final img2code model is trained on 8.6 million image-code pairs and improves accuracy, code quality, and rendered-image quality over successive training cycles.
  • Using temperature 0.7, Img2Code-8B synthesizes more diverse and creative images from 1.57 million raw foundational-dataset images.
  • The synthesis results include examples of low-quality outputs such as blank images, random shapes, black squares, and externally dependent images.
  • A prompt converts synthesized images into concise K-12 math reasoning questions by analyzing patterns, shapes, and numerical relationships.
  • The evaluation includes comparisons between original, initial-model, and final-model outputs, alongside tables assessing translated and image-to-code performance.
  • New synthesized-image examples are organized across K12-2M, arXiv, and MathV360k source collections.
Loading 2505.10557v1…