Source-linked AI summary
MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
Xinyan Chen, Renrui Zhang, Dongzhi Jiang, Aojun Zhou, Shilin Yan, Weifeng Lin, Hongsheng Li
TL;DR
Visual mathematical reasoning remains challenging because existing approaches have limitations in textual-only reasoning, region selection, vision-encoder perception, and image modification. MINT-CoT selects visual tokens for each reasoning step and trains with token-aligned data; it reports improvements over a baseline on three benchmarks.
Problem
Visual mathematical reasoning remains challenging because text-only CoT can misinterpret images, box-based selection can be too coarse, vision encoders can struggle with math content, and some methods depend on external visual modification.
Source: §1 · pp. 1–3Method
MINT-CoT uses an Interleave Token to select relevant visual tokens for each reasoning step, trains on a 54K token-aligned dataset, and applies a three-stage training strategy.
Source: §1 · p. 3Results
+32.59% on MathVista mathematical subsets, +26.92% on GeoQA, and +23.2% on MMStar-Math are reported over the baseline.
Source: §4.2 · p. 8Takeaways & Limitations
Experiments with MINT-CoT-7B demonstrate improvements across various benchmarks for reasoning over interleaved visual tokens.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) has widely enhanced mathematical reasoning in Large Language Models (LLMs), but it still remains challenging for extending it to multimodal domains. Existing works either adopt a similar textual reasoning for image input, or seek to interleave visual signals into mathematical CoT. However, they face three key limitations for math problem-solving: reliance on coarse-grained box-shaped image regions, limited perception of vision encoders on math content, and dependence on external capabilities for visual modification. In this paper, we propose MINT-CoT, introducing Mathematical INterleaved Tokens for Chain-of-Thought visual reasoning. MINT-CoT adaptively interleaves relevant visual tokens into textual reasoning steps via an Interleave Token, which dynamically selects visual regions of any shapes within math figures. To empower this capability, we construct the MINT-CoT dataset, containing 54K mathematical problems aligning each reasoning step with visual regions at the token level, accompanied by a rigorous data generation pipeline. We further present a three-stage MINT-CoT training strategy, progressively combining text-only CoT SFT, interleaved CoT SFT, and interleaved CoT RL, which derives our MINT-CoT-7B model. Extensive experiments demonstrate the effectiveness of our method for effective visual interleaved reasoning in mathematical domains, where MINT-CoT-7B outperforms the baseline model by +34.08% on MathVista, +28.78% on GeoQA, and +23.2% on MMStar, respectively. Our code and data are available at https://github.com/xinyan-cxy/MINT-CoT
1 Introduction
MINT-CoT addresses limitations in visual mathematical reasoning by selecting relevant image tokens for each reasoning step, supported by a token-aligned dataset and a progressive training strategy. The paper reports improvements over a baseline on MathVista, GeoQA, and MMStar.
1 Introduction
- Existing approaches face coarse box-shaped regions, limited vision-encoder perception of math content, and reliance on external visual-modification capabilities.Box selection can include irrelevant or misleading tokens in interconnected mathematical images; encoders trained on natural images may struggle with math images, while modification approaches can require training data or extra inference tools.Source: §1 · pp. 2–3
- MINT-CoT uses an Interleave Token to select relevant visual tokens for each reasoning step, allowing regions such as line segments and coordinates rather than only boxes.Selection uses similarity scores between the Interleave Token hidden state and visual tokens.Source: §1 · p. 3
- Its 54K dataset aligns reasoning steps with selected image-token indices using a pipeline of grid division, OCR localization, keyword extraction, and visual-region assignment.The source mathematical problems come from Mulberry-260K.Source: §1 · p. 3
- +32.59% on MathVista, +26.92% on GeoQA, and +23.2% on MMStar are reported over the baseline for MINT-CoT-7B.The reported results accompany a three-stage training strategy.Source: §1 · p. 3
- The MINT-CoT training strategy progresses through text-only CoT training, interleaved CoT SFT, and interleaved CoT RL.The strategy is presented as incrementally improving MLLMs’ ability to perform interleaved mathematical reasoning.Source: §1 · p. 3
2 Related work
Prior work on visual mathematical reasoning includes data-focused approaches and two visual-CoT strategies: textual reasoning over images or explicit incorporation of visual elements. MINT-CoT’s framework selects visual tokens for each reasoning step.
2 Related work
- Math-focused MLLM work includes dataset expansion, mathematical vision-encoder adaptation, and geometry-image generation, as in G-LLaVA [15], Math-LLaVA [57], MAVIS, and R-CoT [9].The cited passage describes these as approaches to enhancing visual mathematical reasoning.Source: §2 · pp. 3–4
- MINT-CoT generates an Interleave Token before each reasoning step and selects visual tokens using similarity scores between projected embeddings.The selected visual tokens are used for inference with the reasoning step.Source: Figure 2 · p. 4
- Visual CoT methods use textual analysis or incorporate image regions and auxiliary diagrams, while prior methods are limited by rigid cropping, external tools, or insufficiently informative attention scores.The passage names R1-V, Visual CoT [55], Chain-of-Spot [42], Chain-of-Image [47], Visual SKETCHPAD [26], and ICoT.Source: §2 · p. 4
3 Method
MINT-CoT interleaves selected visual tokens into mathematical reasoning steps, supported by a 54K-sample annotated dataset and a progressive three-stage training strategy.
3.1 MINT-CoT
- The Interleave Token selects visual tokens relevant to each step’s mathematical concepts, using scaled cosine-similarity scores and a threshold.Selected tokens are interleaved before the corresponding textual step, grounding visual evidence throughout the reasoning chain.Source: §3.1 · p. 5
3.2 Dataset Curation
- The 54K-sample dataset aligns reasoning steps with visual grid indices through image gridding, OCR, keyword extraction, and GPT-4o annotation.It draws on mathematical problems and reference reasoning steps from Mulberry-260K.Source: §3.2 · pp. 5–6, Figure 3 · p. 6
3.3 Training strategy
- Training progresses from text-only CoT SFT to interleaved CoT SFT and then interleaved CoT RL, producing MINT-CoT-7B.Text-only SFT establishes a general reasoning format; interleaved SFT trains visual-token selection, and RL uses answer-correctness rewards with GRPO.Source: §3.3 · pp. 6–7
4 Experiments
Experiments evaluate MINT-CoT-7B on mathematical benchmarks and use training-stage and interleaving ablations, alongside qualitative results, to examine its performance.
4.1 Experimental Settings
- Evaluation covers GeoQA's Geo170K test set, MathVista's mathematical subsets, and the mathematical capability dimension of MMStar.MathVista-Math covers geometry reasoning, algebraic reasoning, geometry problem solving, and textbook question answering.Source: §4.1 · pp. 7–8
4.2 Quantitative Results
- MINT-CoT-7B improves over the baseline by +32.59% on MathVista, +26.92% on GeoQA, and +23.2% on MMStar-Math.The evaluation uses MathVista's mathematical subset; the reported comparisons are against the baseline model.Source: Table 1 · p. 8, Table 2 · p. 8, Table 3 · p. 8
4.3 Ablation Study
- Original Image CoT SFT underperforms on MathVista-Math and GeoQA, while Bounding Box CoT SFT trails token selection except on TQA and falls below text-only on MathVista-Math GEO and GPS.The authors attribute the original-image decline as likely due to interleaving excessive unrelated visual tokens.Source: Table 5 · p. 9, §4.3 · pp. 9–10
- Text-only CoT SFT adds +21.2%, +21.22%, and +22.96% across MMStar-Math, GeoQA, and MathVista-Math; Interleaved CoT SFT adds +0.4%, +3.05%, and +3.71%, and RL adds +1.6%, +2.65%, and +5.92%, respectively.The reported increments are for the three benchmarks in that order at each training stage.Source: §4.3 · p. 9, Table 4 · p. 9
4.4 Qualitative Results
- Figure 5 shows more coherent reasoning and relevant visual-token interleaving; during Interleaved CoT SFT, selected-token F1 trends upward with fluctuations, while RL F1 is not reported without online ground-truth token indices.The F1 plot tracks selected visual tokens against ground-truth tokens at each reasoning step.Source: Figure 4 · p. 9, Figure 5 · p. 10, §4.4 · p. 10
5 Conclusion
The conclusion presents MINT-CoT as a method for interleaving fine-grained visual tokens into mathematical chain-of-thought reasoning. It also summarizes the dataset and training strategy and reports improvements across benchmarks.
5 Conclusion
- MINT-CoT uses an Interleave Token to automatically select fine-grained visual tokens for each reasoning step.The method is presented as enhancing multimodal mathematical reasoning.Source: §5 · p. 10
- The MINT-CoT dataset is introduced alongside a four-step generation pipeline for visual interleaved reasoning data.Source: §5 · p. 10
- The three-stage training strategy combines Text-only CoT Training, Interleaved CoT SFT, and Interleaved CoT RL, and MINT-CoT-7B improves across benchmarks.Source: §5 · p. 10
A.1 Overview
The supplementary material is organized into sections on dataset details, theoretical and implementation details, ablations, and qualitative results.
A.1 Overview
- The supplementary material includes dataset examples and statistics.Source: §A.1 · p. 18
- It also provides theoretical details of Interleaved CoT RL and additional implementation details.Source: §A.1 · p. 18
- Further material covers a projector ablation, additional ablation studies, and qualitative results.Source: §A.1 · p. 18
A.2 Dataset Details
The MINT-CoT dataset comprises 54,031 data points from the mathematical portion of Mulberry-260k, with examples illustrating its visual and textual annotations.
A.2 Dataset Details
- Figures 6 to 8 show dataset examples, with yellow highlights marking interleaved grid indices and blue highlights marking key words in each reasoning step.Source: §A.2 · p. 18
- 54,031 MINT-CoT data points are derived from the mathematical portion of Mulberry-260k.Table 6 presents key dataset statistics.Source: §A.2 · p. 18, Table 6 · p. 18
- The dataset statistics are presented in Table 6.Source: Table 6 · p. 18
A.3 Theoretical Details of Interleaved CoT RL
The Interleaved CoT RL objective combines losses for textual tokens and visual-token selection. Its visual loss uses scaled similarity scores from interactions between Interleave tokens and input visual tokens.
A.3 Theoretical Details of Interleaved CoT RL
- The textual-token loss uses a reference policy as a regularization target, with KL divergence penalizing deviation to encourage stable updates.Source: §A.3 · p. 18
- The visual-token loss is applied to scaled similarity scores derived from interactions between Interleave tokens and input visual tokens across reasoning steps.The passages define the number of steps and the number of visual tokens interleaved per step.Source: §A.3 · p. 18
- The projector ablation compares configurations without projectors, with single-layer linear projections, and with two-layer MLPs.Source: Table 7 · p. 19
- The final GRPO policy loss sums the textual-token loss and a weighted visual-token loss.The visual loss is rescaled by λ.Source: §A.3 · p. 19
A.4 Additional Implementation Details
MINT-CoT-7B is trained in three stages, progressing from text-only CoT to interleaved CoT SFT and RL. The appendix also specifies the training setup and two comparison baselines.
A.4 Additional Implementation Details
- Training progresses through text-only CoT training, interleaved CoT SFT, and interleaved CoT RL to produce MINT-CoT-7B.The stages run for 2 epochs, 3 epochs, and 700 steps, respectively.Source: §A.4 · p. 19
- The vision encoder remains fixed while all other model parameters, including the Interleave Token and projector layers, are unfrozen during training.Source: §A.4 · p. 19
- Bounding Box CoT SFT uses the minimal enclosing rectangle around selected-token labels, while Original Image CoT SFT prepends the entire image to each step.The bounding-box baseline trains for 2 epochs; the full-image baseline trains for 1 epoch.Source: §A.4 · p. 19
A.5 Additional Ablation Study
The projector ablation reports that single-layer linear projectors perform best among the tested configurations. Additional qualitative figures compare Qwen2-VL-7B-Instruct with MINT-CoT-7B and include a geometry problem example.
A.5 Additional Ablation Study
- Single-layer linear projectors perform best over the tested projector configurations on the mathematical subset of MathVista.The alternatives remove both projectors or replace them with two-layer GELU MLPs; the study applies to the Interleaved CoT SFT stage.Source: §A.5 · p. 19
A.6 Additional Qualitative Results
- Figures 9, 10, and 11 compare Qwen2-VL-7B-Instruct with MINT-CoT-7B, without a stated outcome in the supplied captions.Source: Figure 9 · p. 21, Figure 10 · p. 21, Figure 11 · p. 22
- One qualitative example asks for the perimeter of triangle CDE given a 200 cm perimeter for ABCD and a perpendicular-bisector construction.Source: §A.6 · p. 22