Source-linked AI summary
CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Images
Chengqi Duan, Kaiyue Sun, Rongyao Fang, Manyuan Zhang, Yan Feng, Ying Luo, Yufang Liu, Ke Wang, Peng Pei, Xunliang Cai, Hongsheng Li, Yi Ma, Xihui Liu
TL;DR
Mathematical problems requiring visual assistance expose limitations in text-only reasoning and imprecise image generation. CodePlot-CoT represents visual thoughts as executable plotting code, supported by the Math-VR benchmark and MatplotCode converter, and achieves up to 21% improvement over base models. The work provides a code-driven approach for multimodal mathematical reasoning, although image-to-code fidelity remains imperfect.
Problem
Mathematical problems requiring auxiliary lines or plotted functions remain difficult because text-only reasoning is limited and unified multimodal models lack precision and controllability.
Method
CodePlot-CoT generates executable plotting code, renders it into images as visual thoughts, and feeds those images back into VLM reasoning, using Math-VR and MatplotCode for training data.
Results
Up to 21% improvement over base models is reported on the mathematical visual-reasoning benchmark.
Takeaways & Limitations
The paper introduces a code-driven paradigm, bilingual benchmark, and image-to-code converter for mathematical visual reasoning.
Takeaways & Limitations
Image-to-code conversion has not reached 100% fidelity, so generated visual reasoning can remain slightly imperfect.
Abstract
from arXiv · showhide
Recent advances in Large Language Models (LLMs) and Vision Language Models (VLMs) have shown significant progress in mathematical reasoning, yet they still face a critical bottleneck with problems requiring visual assistance, such as drawing auxiliary lines or plotting functions to solve the problems. Most LLMs and VLMs are constrained to text-only reasoning chains, while multimodal unified models that can generate interleaved text and images lack the necessary precision and controllability for such tasks. To address this, we propose CodePlot-CoT, a code-driven Chain-of-Thought paradigm for "thinking with images" in mathematics. Our approach leverages the VLM to generate text reasoning as well as executable plotting code, which is then rendered into images as "visual thought", to solve mathematical problems. To achieve this, we first construct Math-VR, the first large-scale, bilingual dataset and benchmark for Mathematics problems with Visual Reasoning, comprising 178K samples. Second, to create high-quality training data, we develop a state-of-the-art image-to-code converter specialized for parsing complex mathematical figures into codes. Finally, using these training data, we train the CodePlot-CoT model for solving mathematical problems. Experimental results show that our model achieves up to 21% increase over base model on our new benchmark, fully validating the efficacy of our proposed code-driven reasoning paradigm. Our work opens a new direction for multimodal mathematical reasoning and provides the community with the first large-scale dataset, comprehensive benchmark, and strong approach for such problems. To facilitate future research, we make our datasets, code, and pretrained models publicly available at https://github.com/HKU-MMLab/Math-VR-CodePlot-CoT.
1 INTRODUCTION
The paper targets mathematical problems requiring visual reasoning, where text-only chains and imprecise image generation are inadequate. It proposes code-driven visual thoughts and introduces the Math-VR benchmark, MatplotCode converter, and CodePlot-CoT model.
- 1 INTRODUCTION: Math-VR requires deep visual reasoning, unlike existing benchmarks that rely on textual reasoning.The benchmark addresses mathematical problems where visual assistance is needed during problem solving.
- 1 INTRODUCTION: Text-only reasoning chains and unified multimodal models struggle with precise geometric operations such as constructing constrained auxiliary lines.Direct image generation is insufficient for the precision required by mathematical visual reasoning.
- 1 INTRODUCTION: CodePlot-CoT generates executable plotting code whose rendered images serve as intermediate visual thoughts in the reasoning sequence.The images are fed back into the VLM after code execution.
- 1 INTRODUCTION: The paper constructs Math-VR, a 178K-sample bilingual dataset and benchmark for mathematical visual reasoning.The work also develops MatplotCode, an image-to-code converter for mathematical figures.
- 1 INTRODUCTION: Up to 21% performance increase over strong baselines is achieved by CodePlot-CoT for code-driven visual reasoning.The model is trained using the paper’s proposed visual-reasoning data and MatplotCode converter.
2 RELATED WORK
Prior mathematical visual-reasoning systems use interleaved image manipulation or code-based tools, but precise and interpretable visual control remains limited. These limitations motivate a code-driven approach for mathematical visual reasoning.
- “Thinking with image” Models: “Thinking with image” methods retrieve, crop, or generate visual aids and interleave them with multimodal reasoning.These approaches address limitations of text-only reasoning chains.
- Visual Reasoning Models in Mathematics: Interleaved visual-reasoning approaches can add auxiliary lines and plot functions, but their visual actions are weakly controllable.The resulting lack of control hampers precise geometric constructions and limits interpretability of intermediate reasoning.
- Visual Reasoning Models in Mathematics: Mathematical visual-reasoning research predominantly follows interleaved image reasoning and agent-plus-code tool-use paradigms.The paper positions its approach relative to both lines of work.
3 MATH-VR: DATASET AND BENCHMARK FOR MATH VISUAL REASONING
Math-VR is a bilingual benchmark designed to test active visual reasoning with mathematical images rather than text-only solution generation. It combines curated visual-reasoning problems with an evaluation pipeline that separately measures final-answer correctness and progress through required solution steps.
- Dataset Motivation: Math-VR requires reasoning in both text and image domains, unlike benchmarks whose solutions can be completed entirely in text.Its problems may require considering visual scenarios and actively using images during problem solving.
- Dataset Construction: The dataset is built by filtering 900k image-containing problems to retain samples requiring mathematical figures for reasoning, then standardizing questions and solutions.The collection process uses VLM filtering and converts textual images into readable text.
- Dataset Statistics: Math-VR contains questions, reasoning processes, and final answers with visual reasoning, spanning Geometry, Algebra, Calculus, and Statistics.The supplied statistics passage reports 29% text-only and 71% multimodal questions, with Geometry dominant at 81%.
- Evaluation: The evaluation pipeline uses Answer Correctness for binary final-answer judgment and Process Score for partial credit based on matched scoring points.GPT-4.1 extracts reference answers and weighted scoring points from each ground-truth solution; completely correct answers receive full Process Score.
- Benchmark Statistics: The benchmark is divided into 2k text-only and 3k multimodal questions, and its distribution includes Geometry as the dominant knowledge type at 77%, with Algebra and Calculus comprising 13%.The benchmark statistics passage defines the Text and Multimodal subsets; Figure 3 reports the knowledge-type distribution.
4 CODEPLOT-COT PARADIGM: CODE-DRIVEN COT FOR MATHEMATICS VISUAL REASONING
CodePlot-CoT replaces imprecise direct image generation with executable plotting code that produces visual thoughts interleaved with language reasoning. The approach is supported by MatplotCode, a specialized image-to-code converter used to create code–image training data.
- CodePlot-CoT Paradigm: CodePlot-CoT represents visual thoughts as executable plotting code rather than pixel-encoded images, targeting precise geometric shapes, positions, and angular relations.The model writes plotting code, which is executed to render mathematical figures for subsequent reasoning.
- CodePlot-CoT Paradigm: The model interleaves natural-language reasoning with generated plotting code, renders the code into an image, and feeds that visual thought back into the reasoning sequence.This supports multi-image reasoning and self-generated visual evidence when a step requires visual support.
- Image-to-Code Data Curation: MatplotCode addresses the lack of paired mathematical figure, plotting-code, and reasoning-chain data by converting complex mathematical figures into code for scalable training-data creation.Existing mathematical resources rarely provide code annotations, and zero-shot conversion from complex figures is described as unreliable.
- Image-to-Code Data Curation: MatplotCode is trained on a curated subset of ImgCode-8.6M focused on geometry diagrams and function plots, with Python used for code representations.The supplied passage states that the converter demonstrates superior generalization and conversion fidelity on this curated data.
- Training: The CodePlot-CoT model is initialized from vision-aligned MatPlotCode and fully fine-tuned on a curated supervised dataset for 5000 steps.MatPlotCode itself uses vision alignment followed by full-parameter fine-tuning before CodePlot-CoT initialization.
5 EXPERIMENTS
Experiments evaluate CodePlot-CoT on Math-VR, its image-to-code converter, inference cost, and ablations against text-only and direct image-generation reasoning.
- 5 EXPERIMENTS: Human-correlation checks report Cohen’s κ = 0.75 and MCC = 0.75 for answer correctness, alongside Pearson r = 0.72 and Spearman ρ = 0.70 for process scores.The study uses 15 STEM undergraduates judging 3,000 answers sampled from 1,000 benchmark questions.
- 5.2 BENCHMARKING EXISTING MODELS AND CODEPLOT-COT ON MATH-VR: Gemini-2.5-Pro achieves the highest overall baseline scores, with PS = 80.8 and AC = 64.7, but still fails on about one-third of problems.The benchmark compares state-of-the-art LLMs, VLMs, and unified models on 2,500 English questions.
- 5.2 BENCHMARKING EXISTING MODELS AND CODEPLOT-COT ON MATH-VR: Up to 21% improvement over the 32B base VLM, with CodePlot-CoT largely outperforming Qwen2.5-VL-72B across metrics.The comparison attributes the gains to structured, verifiable visual reasoning rather than model size or longer textual chains.
- 5.3 IMAGE-CODE CONVERTER EVALUATION: MatplotCode reaches 100% execution success and is preferred in 554 of 1,000 reconstruction comparisons, ahead of FigCodifier-8B, GPT-o3, and Gemini-2.5-Pro.The converter evaluation measures both executable-code reliability and reconstruction fidelity judged by GPT-4.1.
- 5.4 ANALYSIS ON INFERENCE COST: The model generates 820.9 tokens per image and averages 1.37 rendered images per problem, with local rendering taking less than one second.The reported output length is lower than many autoregressive thinking-with-image models using 1,024 to 4,096 tokens per image.
- 5.5 ABLATION STUDIES: CodePlot-CoT outperforms text-only fine-tuning and direct image generation in ablations, supporting executable code as a more precise and controllable visual-reasoning representation.The comparisons use Qwen-2.5VL-3B for text-only reasoning and Bagel for direct image generation.
6 CONCLUSION
The paper introduces CodePlot-CoT for mathematical visual reasoning by representing visual thoughts as executable plotting code, supported by Math-VR and MatplotCode. Experiments report improvements of up to 21% over baseline models.
- 6 CONCLUSION: CodePlot-CoT represents mathematical visual reasoning as executable plotting snippets, supported by the Math-VR benchmark and MatplotCode image-to-code converter.The approach is presented as a way to produce precise and controllable visual thoughts.
7 ETHICS AND REPRODUCIBILITY STATEMENT
The paper states that its data come from publicly available websites and contain no private or sensitive information. Dataset, benchmark, training, evaluation, and verification details are distributed across the main text and appendix.
- 7 ETHICS AND REPRODUCIBILITY STATEMENT: All study data are collected from publicly available websites, with no private or sensitive information used in dataset construction.Further dataset, benchmark, evaluation, training, and manual-verification details are documented in the paper and appendix.
A.2 DATASET CATEGORIZATION
Math-VR contains varied mathematical problems and knowledge types, with geometry dominating the categorized questions. The dataset includes both text-only and text-plus-image questions, all requiring visual mathematical reasoning.
- A.2 DATASET CATEGORIZATION: Approximately 29% of Math-VR questions are text-only and 71% combine text with mathematical images, while both types require visual reasoning.The dataset contains approximately 90k unique questions spanning single- and multipart, multiple-choice, answer-based, and proof-based formats.
- A.2 DATASET CATEGORIZATION: Geometry forms 76% of categorized Math-VR problems, while Algebra and Calculus together account for 12%.The knowledge-type distribution is shown in Figure 6.
- A.2 DATASET CATEGORIZATION: The dataset’s key statistics are summarized in Table 5, covering question structure, answer formats, and image-supported solutions.Answer-based questions are the most prevalent, and each solution contains at least one image on average.
B BENCHMARK
The benchmark uses GPT-4.1 for sample metadata and response evaluation, with manual review to verify sample quality and answers. Training uses Qwen2.5VL-32B-Instruct as the base model for MatPlotCode and CodePlot-CoT.
- B BENCHMARK: GPT-4.1 generates sample meta information and evaluates model responses using dedicated prompting templates.The templates are illustrated in Figures 7 and 8.
- B BENCHMARK: Manual review combines selecting samples requiring non-trivial visual reasoning with verifying their metadata and answers.The review interface presents questions, solutions, and metadata together for annotator checks.
- B BENCHMARK: Qwen2.5VL-32B-Instruct serves as the base model for both MatPlotCode and CodePlot-CoT.The models are trained with separate alignment and fine-tuning procedures on 32 NVIDIA H200 GPUs.
D CODEPLOT-COT SAMPLES ON MATH-VR BENCHMARK
CodePlot-CoT is evaluated on both multimodal and text-only Math-VR questions, while both settings require visual reasoning. The examples indicate strong textual and visual reasoning capabilities from the code-driven approach.
- D CODEPLOT-COT SAMPLES ON MATH-VR BENCHMARK: The examples cover multimodal inputs containing a math figure and purely text-based inputs.Figures 12 and 13 illustrate the two input modalities.
- D CODEPLOT-COT SAMPLES ON MATH-VR BENCHMARK: Both input types require visual reasoning despite differing in whether an image appears in the question.The distinction is between input modality and the visual reasoning demanded by the problem.
- D CODEPLOT-COT SAMPLES ON MATH-VR BENCHMARK: The samples demonstrate strong reasoning capabilities in textual and visual reasoning, supporting the proposed code-driven visual reasoning paradigm.
E LIMITATIONS
The main limitation is imperfect image-to-code conversion caused by constrained data scale and model size. Consequently, some generated visual reasoning images may contain inaccuracies, including slight geometric displacement.
- E LIMITATIONS: Image-to-code conversion has not reached 100% fidelity because of limitations in data scale and model size.
- E LIMITATIONS: Some visual reasoning images are therefore not entirely accurate, which can affect the final model's generated visual reasoning.
- E LIMITATIONS: In Figure 14, the generated image places point H slightly away from edge AD, although H should lie on that edge.