MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning

• Prior VCoT methods rely on rigid external tools, while intrinsic approaches have struggled with accurate diagram generation and strategic decisions about when and how to use visual aids.The paper identifies diagram fidelity and procedural knowledge for using visualizations in deduction as the two key challenges.Source: §1 · p. 2
• MathCanvas combines visual manipulation with strategic visual-aided reasoning to give unified LMMs intrinsic VCoT capabilities for complex mathematical problem-solving.The framework addresses both diagram generation and editing, and the strategic use of diagrams within reasoning.Source: §1 · p. 2
• 15.2M training pairs support visual manipulation: 10M caption-to-diagram pairs and 5.2M step-by-step diagram-editing instruction pairs.The editing pairs are generated through a hybrid pipeline combining LLM-driven mining with programmatic synthesis.Source: §1 · p. 2
• 219K MathCanvas-Instruct examples interleave textual reasoning with corresponding visual steps to teach models how to coordinate diagrams with reasoning trajectories.The dataset is used in the Strategic Visual-Aided Reasoning phase.Source: §1 · p. 2
• MathCanvas-Bench contains 3K problems requiring coherent interleaved reasoning and visual outputs, and the authors benchmarked 20 leading LMMs on it.The benchmark is presented as a challenging testbed for visual–textual mathematical reasoning.Source: §1 · p. 2

Mathematical Reasoning with Large Multimodal Models

• Later multimodal math models remain text-centric, interpreting visual inputs but largely neglecting vision as an active, generative part of reasoning (Yang et al., 2025; Wang et al., 2025b; Duan et al., 2025; Wei et al., 2025).Earlier approaches adapted vision-text alignment and fine-tuned on mathematical question-answer pairs (Gao et al., 2025a; Wang et al., 2025a; Zhuang et al., 2024; Zhang et al., 2024b; Guo et al., 2025).Source: §2 · p. 3

Visual Chain-of-Thought

• External-tool VCoT is constrained by non-generative or rigid tools, while intrinsic VCoT remains demonstrated mainly in simpler domains and struggles with precise, logically consistent diagrams for complex mathematics (Shao et al., 2024a; Chen et al., 2025; Hu et al., 2024; OpenAI, 2025b; Gao et al., 2025b; Wang et al., 2025c,d; Cheng et al., 2025; Li et al., 2025b,c; Chern et al., 2025; Li et al., 2025a).VCoT integrates visual-aid generation into reasoning, unlike textual chain-of-thought (Wei et al., 2023; Fang et al., 2025a,b).Source: §2 · p. 3

Datasets and Benchmarks for Multimodal Mathematical Reasoning

• Existing multimodal math benchmarks use static problem-solution pairs rather than step-by-step visual demonstrations, motivating MathCanvas-Instruct and MathCanvas-Bench (Lu et al., 2021; Lu et al., 2022; Yue et al., 2024; Lu et al., 2024; Wang et al., 2024; Zhang et al., 2024a; Qiao et al., 2024; Wang et al., 2025e; Sun et al., 2024).Foundational datasets established the task, while later challenging benchmarks pushed the limits of LMM visual reasoning.Source: §2 · p. 3

3.1.1 Million-scale Pretraining Corpus

• The pretraining corpus pairs 5.2M diagram-edit trajectories with 10M caption-to-diagram examples.MathCanvas-Edit combines 4.2M competition-derived trajectories with 1M systematically generated trajectories; MathCanvas-Imagen aggregates examples from edit steps, code-derived captions, and public datasets.Source: §3.1.1 · p. 4

3.1.2 MathCanvas-Instruct

• MathCanvas-Instruct provides 219K examples of interleaved visual-textual mathematical reasoning for training.After filtering an initial 632K problems, deduplication yielded 222K unique pairs; images were enhanced and standardized, and problems were classified into eight major categories and fine-grained subcategories.Source: §3.1.2 · p. 5

3.2 The MathCanvas-Bench Evaluation Benchmark

• MathCanvas-Bench samples 3K problems, excludes multiple-choice questions, balances categories, and removes near-duplicates from the training set.GPT-4.1 extracts and checks final answers; evaluation uses complete accuracy and weighted scoring that gives exponentially increasing weights to sub-questions.Source: §3.2 · pp. 5–6

3.3 Two-Stage Training Recipe

• MathCanvas trains BAGEL in two stages: visual manipulation followed by strategic visual-aided reasoning.Stage I trains the Generation Expert on MathCanvas-Edit and MathCanvas-Imagen; Stage II fine-tunes the entire model on MathCanvas-Instruct.Source: Figure 4 · p. 6, §3.3 · p. 6
• Stage I trains the Generation Expert on 5.2M edit trajectories and 10M image-caption pairs, while freezing the understanding pathway.Edit trajectories contain sequences of 2–4 diagram transformations, and diagram generation is trained with Rectified-Flow Loss (Liu et al., 2022).Source: §3.3 · p. 6

Stage II: Strategic Visual-Aided Reasoning

• Stage II jointly fine-tunes all model components to choose when to draw and to interleave generated diagrams with textual reasoning.After each text segment, token prediction selects whether to begin drawing or conclude; question images provide clean VAE and ViT context, while solution images also use noised VAE tokens for Rectified-Flow Loss.Source: Stage II: Strategic Visual-Aided… · pp. 6–7, Figure 4 · p. 6

4 Experiments

• BAGEL-Canvas is compared against 20 prominent LMMs, with evaluations conducted using VLMEvalKit for a fair comparison.The comparisons include proprietary and open-source models, BAGEL, and BAGEL-Zebra-CoT.Source: §4 · p. 7

4.1 Benchmark Results

• BAGEL-Canvas scores 34.4% weighted on MathCanvas-Bench, leading open-source models and improving 15.9 points over BAGEL.It surpasses Qwen-2.5-VL-72B (32.8) and InternVL3.5-30B-A3B (22.2), and also outperforms Gemini-2.0-Flash (32.6) and GPT-4.1 (30.0).Source: §4.1 · p. 7
• BAGEL-Canvas improves over BAGEL most in Trigonometry (+27.1), Plane Geometry (+19.2), and Solid Geometry (+12.3), but gains only +0.8 in Calculus & Vector.It also gains +14.1 in Analytic Geometry and +11.8 in Algebra; the paper says Calculus & Vector may require specialized reasoning beyond its visual augmentation techniques.Source: §4.1 · pp. 7–8

4.2 Performance on Other Math Benchmarks

• On three public text-only benchmarks, BAGEL-Canvas improves over BAGEL, including MathVerse (+17.9) and MathVista (+10.5).On MathVision, gains include Analytic Geometry (+22.6), Algebra (+13.0), and Metric Geometry—Angle (+17.3); these results are on benchmarks requiring text-only solutions.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablation Studies

• Removing MathCanvas-Edit lowers weighted score by 2.4 points, and removing the entire pretraining stage lowers it by a further 1.2 points.The findings support contributions from both step-by-step diagram editing and foundational diagram generation.Source: §4.3 · p. 8, Table 3 · p. 8
• Training without visual data lowers weighted score by 3.5 points, while skipping image generation at inference performs 1.0 point better than BAGEL-Canvas-Text.Both ablations reduce performance; the comparison suggests interleaved visual-textual training also enhances textual reasoning.Source: §4.3 · pp. 8–9, Table 4 · p. 8

• 86% relative improvement over strong baselines on MathCanvas-Bench was achieved by BAGEL-Canvas under the MathCanvas training framework.Source: §5 · p. 9
• MathCanvas uses MathCanvas-Edit, MathCanvas-Imagen, and MathCanvas-Instruct in a two-stage recipe to teach BAGEL-Canvas diagram manipulation and strategic interleaving with textual deduction.Source: §5 · p. 9
• The training paradigm enhances BAGEL-Canvas's core textual reasoning and provides a foundation for research into broader and more complex multimodal reasoning.Source: §5 · p. 9

Training setup

• Experiments use BAGEL-7B-MoT (Deng et al., 2025a), AdamW (Loshchilov and Hutter, 2019), and a cluster of 16 NVIDIA H800 GPUs.AdamW is used for both training stages.Source: A Training Details · p. 12

Stage I

• Stage I freezes the understanding expert and trains only the generation expert with Rectified-Flow loss, using 0.3 ViT condition dropout to regularize visual features.No Cross-Entropy loss is used in this stage.Source: A Training Details · p. 12

Stage II

• Stage II unfreezes all components and combines Cross-Entropy loss weighted 0.25 with Rectified-Flow loss weighted 1.0, while lowering ViT condition dropout to 0.1.The learning rate is halved and training steps are reduced for fine-tuning.Source: A Training Details · p. 12

B.1 MathCanvas-Edit and MathCanvas-Imagen

• The MathCanvas-Edit pipeline randomly and incrementally adds geometric primitives and relations to build progressively more complex diagrams.It samples from predefined sets of objects and constructions.Source: §B.1 · p. 12
• The 18-object primitive set includes segments and angles, triangle variants, quadrangles, and pentagons.Named variants include isosceles, right, and equilateral triangles; rectangles, squares, trapezoids, and equal-diagonal quadrangles; and equilateral pentagons.Source: §B.1 · p. 12
• The predefined construction set contains 41 relations, categorized by how many new points they introduce.The passage lists 37 one-point relations and four two-point relations.Source: §B.1 · p. 12
• The two-point relation set comprises square, trisegment, trisect, and tangent constructions.These are among the relations randomly sampled to build diagrams with systematic coverage of fundamental geometric operations.Source: §B.1 · p. 12
• The appendix presents MathCanvas-Edit examples in Figure 6 and MathCanvas-Imagen examples in Figures 7 and 8.Source: §B.1 · p. 12

B.2 MathCanvas-Instruct

• MathCanvas-Instruct contains 219K problems, comprising 65% multimodal and 35% text-only examples.The paper also analyzes problem sources, question and solution lengths, image counts, and knowledge-point distribution.Source: §B.2 · p. 12

C.1 Weighted Scoring Weights

• A 1.3 exponential growth factor gives later sub-questions greater weight in the weighted scoring metric.Source: §C.1 · pp. 12–13
• A problem’s final score sums the weights of correctly answered sub-questions, using normalized weights for problems with up to four sub-questions.Source: §C.1 · p. 13
• For 2 sub-questions, the normalized weights are [0.4348, 0.5652].Source: §C.1 · p. 13
• For 3 sub-questions, the normalized weights are [0.2506, 0.3258, 0.4236].Source: §C.1 · p. 13
• For 4 sub-questions, the normalized weights are [0.1616, 0.2101, 0.2732, 0.3551].Source: §C.1 · p. 13

C.2 Evaluation Template

• Tables 7 and 8 present the prompt templates used to evaluate MathCanvas-Bench.Source: §C.2 · p. 13

D Additional Qualitative Results

• The section provides additional qualitative results comparing BAGEL-Zebra-CoT, Nano-Banana, and MathCanvas.These comparisons are presented in Figure 13.Source: D Additional Qualitative Results · p. 13, Figure 13 · p. 17
• The examples focus on problems that benefit from visual manipulation.The section uses qualitative examples to illustrate limitations of LMMs without intrinsic VCoT capabilities.Source: D Additional Qualitative Results · p. 13
• Figure 12 shows Gemini-2.5-Pro and GPT-5 using complex, sometimes flawed algebra on the problem from Figure 1, contrasting with geometric solutions enabled by intrinsic VCoT.The examples illustrate limitations of leading LMMs when they lack intrinsic visual chain-of-thought capabilities.Source: D Additional Qualitative Results · p. 13, Figure 12 · p. 16