PaperlayerView exact revision on arXiv

Source-linked AI summary

MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems

Shuhang Chen, Hangjie Yuan, Yunqiu Xu, Pengwei Liu, Tao Feng, Jun Cen, Zeying Huang, Yi Yang

arXiv:2503.16549v2cs.CV

TL;DR

MLLMs still struggle to perceive diagrams in visual mathematical problems, and unreliable visual information can hamper later reasoning. The paper introduces FlowVerse and a modular perception–inference pipeline with a dedicated perception model; MathFlow-P-7B improves performance when integrated with different inference models.

  • Problem

    MLLMs often struggle to interpret mathematical images, and unreliable visual information can hamper coherent solutions to complex problems.

    Source: §1 · p. 1

  • Method

    The paper introduces FlowVerse to categorize problem information and MathFlow to decouple perception from inference, including MathFlow-P-7B for extracting diagram EI and RP.

    Source: §1 · pp. 2–3

  • Results

    MathFlow-P-7B yields superior performance improvements when integrated with different inference models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify robust perception as critical for deriving useful properties from visual mathematical diagrams.

    Source: §1 · p. 2

    FlowVerse is mainly available in English and Chinese, limiting its applicability to a wider range of languages.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Despite strong results on many tasks, multimodal large language models (MLLMs) still underperform on visual mathematical problem solving, especially in reliably perceiving and interpreting diagrams. Inspired by human problem-solving, we hypothesize that the ability to extract meaningful information from diagrams is pivotal, as it directly conditions subsequent inference. Hence, we introduce FlowVerse, a comprehensive benchmark that provides a fine-grained evaluation of MLLMs' perception and reasoning capabilities. Our preliminary results on FlowVerse reveal that existing MLLMs exhibit substantial limitations when extracting essential information and reasoned properties from diagrams and performing complex reasoning based on these visual inputs. In response, we introduce MathFlow, a modular problem-solving pipeline that decouples perception and inference into distinct stages, thereby optimizing each independently. Given the perceptual limitations observed in current MLLMs, we trained MathFlow-P-7B as a dedicated perception model. Experimental results indicate that MathFlow-P-7B yields substantial performance gains when integrated with various closed-source and open-source inference models. This demonstrates the effectiveness of the MathFlow pipeline and its compatibility with diverse inference frameworks. Project page: https://github.com/MathFlow-zju/MathFlow.

1 Introduction

MLLMs remain weak at visual mathematical problems, particularly at perceiving diagrams, and unreliable extracted information can hamper subsequent reasoning. The paper introduces FlowVerse to evaluate perception and reasoning, and MathFlow to separate perception from inference, including a model specialized for diagram information extraction.

1 Introduction

  • FlowVerse categorizes problem information as DI, EI, OQ, and RP, then evaluates MLLMs across six variants that combine these information types.The first three categories come from the original problem statement; RP is extracted from the solution.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • MathFlow separates perception and inference: it extracts EI, derives RP, converts them into text, and concatenates them with the original problem text for inference.The resulting unified input combines visual-derived and textual information.Source: §1 · pp. 2–3
  • Preliminary FlowVerse results show that MLLMs struggle to extract EI and RP from diagrams and to perform complex reasoning on images.The paper identifies unreliable perception as a factor that bounds overall problem-solving performance.Source: §1 · p. 2
  • MathFlow-P-7B targets extraction of EI and RP from diagrams, using multi-task pretraining on EI and RP captioning followed by supervised fine-tuning.The authors report performance improvements when it is integrated with different inference models.Source: §1 · p. 2
  • MathFlow-P-7B demonstrates state-of-the-art performance across various benchmarks and improves performance when integrated with different inference models.The authors report that these improvements underscore the effectiveness of the MathFlow pipeline.Source: §1 · pp. 2–3

2 Related Work

Related work addresses mathematical reasoning, visual comprehension, and evaluation benchmarks, while the authors identify inaccurate visual comprehension as a major bottleneck in visual mathematical reasoning.

Visual Mathematical Reasoning with MLLMs

Mathematics Evaluation Benchmarks

  • Mathematics-specific benchmarks assess LLM and MLLM mathematical problem solving, with newer work increasingly addressing problems containing both textual and visual content [Liu et al., 2024c; Satpute et al., 2024; Zhang et al., 2024; Li et al., 2024].MathVerse builds on MathVista to reduce textual shortcuts and adds a chain-of-thought evaluation strategy [Wei et al., 2022; Wu et al., 2024].Source: §2 · p. 3
  • The authors report that benchmark datasets for visual mathematical problem solving still have limitations, despite approaches designed to test genuine diagram interpretation and assess intermediate reasoning steps.Source: §2 · p. 3

3 FlowVerse Benchmark

FlowVerse evaluates visual-mathematical problem solving through distinct information components and problem versions. Preliminary results highlight difficulties extracting diagram information and the importance of robust perception for deriving useful properties.

3 FlowVerse Benchmark

  • FlowVerse contains 2,000 visual mathematical problems from Chinese and English exams, yielding more than 12,000 test samples.The dataset was reviewed for answer and analysis accuracy and consistency between questions and diagrams.Source: §3 · p. 3
  • FlowVerse separates problem information into Descriptive Information (DI), Essential Information (EI), Reasoned Property (RP), and Only Question (OQ).DI describes depicted elements, EI covers critical values or relationships, RP captures inferred properties, and OQ is the question posed.Source: §3 · pp. 3–4
  • Expert annotators create six problem versions by removing or changing the modality of DI, EI, RP, and OQ, producing 12,000 test instances.The versions range from text-centric inputs to entirely visual inputs, supporting evaluation of perception and its contribution to multimodal reasoning.Source: §3 · p. 4
  • FlowVerse-CoT-E progressively adds expert-decomposed solution steps to prompts and aggregates model scores using a weighted approach.The scoring balances intermediate-step scores with the score based only on the final answer; α defaults to 0.8.Source: §3 · p. 5
  • FlowVerse preliminary experiments highlight the importance of robust perception for deriving useful properties from visual mathematical diagrams.The benchmark also reports difficulty extracting essential information and reasoned properties from diagrams and performing complex reasoning on images.Source: §3 · pp. 3, 5

Mathematical Problem-Solving Pipeline

MathFlow separates visual perception from mathematical inference, passing text-form visual information to an inference model alongside the original problem. Its dedicated perception model is trained through multi-task pretraining and supervised fine-tuning.

Mathematical Problem-Solving Pipeline

  • MathFlow separates perception, which extracts and textualizes critical visual information, from inference, which receives the enriched input and original problem.The perception and inference stages may use different models, allowing other state-of-the-art MLLMs or LLMs to provide inference capability.Source: Mathematical Problem-Solving Pipeline · p. 5
  • MathFlow-P-7B is trained in multi-task pretraining and supervised fine-tuning stages.Pretraining covers essential-information captioning and visual-oriented reasoning; fine-tuning uses MathFlow-SFT while freezing the vision encoder and training the perceiver resampler and LLM backbone.Source: Mathematical Problem-Solving Pipeline · p. 6
  • The essential-information caption task trains the model to describe essential elements directly from visual inputs.It uses visual mathematics image-text pairs, including data from MAVIS and Geo170k.Source: Mathematical Problem-Solving Pipeline · p. 6
  • The visual-oriented reasoning task trains extraction of higher-level abstractions and relationships by predicting solution steps from prior-step context.MathFlow-RP is sampled from educational materials with detailed problem-solving solutions.Source: Mathematical Problem-Solving Pipeline · p. 6
  • During multi-task pretraining, the two tasks are mixed at a 3:1 data ratio while the LLM backbone is frozen.Training focuses on the perceiver resampler and vision encoder module.Source: Mathematical Problem-Solving Pipeline · p. 6

5 Experiments

The experiments evaluate MathFlow and MLLMs on FlowVerse and related tasks, testing input representations, evaluation strategies, error patterns, and pipeline components. Results highlight the role of diagram-derived information and report strong perception and integrated-pipeline performance.

5.1 Experimental Setup

  • Experiments use 650,000 image-text pairs for EI captioning and 130,000 visual-oriented reasoning samples derived from 40,000 problems; evaluation is zero-shot.The 130,000 reasoning samples are constructed by decomposing selected problems’ solutions into steps.Source: §5.1 · pp. 6–7

5.2 Benchmark Results on FlowVerse

  • Most MLLMs improve when visual input is removed (+4.3% for Qwen2-VL-72B), while GPT-4V drops -4.1% when descriptive text is removed.Source: §5.2 · p. 7
  • Most models lose accuracy when diagrams must supply EI or RP, and performance also declines when RP shifts from text to images.The RP removal comparison is Text Limited Version versus Vision Dense Version.Source: §5.2 · pp. 7–8

5.3 MathFlow Analysis

  • MathFlow achieves the highest overall accuracy in reported FlowVerse and MathVerse comparisons, while MathFlow⋆GPT-5 leads on FlowVerse† and MathVerse.Table 4 covers MathVerse and raw FlowVerse†.Source: Table 4 · p. 8, §5.3 · p. 8
  • On a curated subset with human annotations, MathFlow-P-7B achieves Structured F1 of 97.2% on EI extraction and 85.6% on RP prediction.Source: Table 5 · p. 8, §5.3 · p. 9

The FlowVerse-CoT-E Strategy Shows More

5.3 MathFlow Analysis

  • Expert annotations show MathFlow⋆ reduces overall and visual-perception errors versus corresponding base models, but many perception errors remain with DeepSeek-r1.Source: §5.3 · p. 8

5.4 Ablation Studies

  • With GPT-4V inference, MathFlow outperforms other configurations; MathFlow⋆DeepSeek-r1 reaches peak performance, while tested alternative perception models underperform MathFlow-P-7B.MathFlow also enables state-of-the-art LLMs to solve visual mathematical problems without additional training.Source: Table 6 · p. 9, Table 7 · p. 9, §5.4 · p. 9

6 Conclusion

FlowVerse exposes perception and visual-reasoning weaknesses in MLLMs; MathFlow separates perception from inference and uses a two-stage-trained perception model.

6 Conclusion

  • FlowVerse reveals limitations in MLLMs’ ability to accurately perceive and process visual information.The benchmark investigates bottlenecks in visual mathematical problem-solving.Source: §6 · p. 9
  • MathFlow decouples problem-solving into distinct perception and inference stages.Source: §6 · p. 9
  • MathFlow-P-7B was developed using a two-stage training strategy to address poor perception performance.Source: §6 · p. 9

Limitations

The limitations section identifies missing difficulty-level categorization and restricted language coverage. The appendix describes FlowVerse’s curation, problem versions, subject coverage, and dataset statistics.

Limitations

  • FlowVerse lacks difficulty-level categorization, which the authors identify as a future direction for deeper model-performance assessment.The current categories reflect subjects and subfields with varying multimodal content.Source: Limitations · p. 10
  • Because FlowVerse is mainly available in English and Chinese, its applicability to a wider range of languages is limited.Source: Limitations · p. 10

A.1 Data Curation

  • After collecting about 2,000 visual math problems, human annotators reviewed answers and diagrams, removed unsuitable items, and retained 2,000 curated problems.Proof-based questions were reformulated as question-and-answer items; problems with minimal Descriptive Information were excluded.Source: §A.1 · p. 14

A.2 Subject and Subfield Definition

  • FlowVerse categorizes problems across plane geometry, solid geometry, and functions, with twelve fine-grained categories within each subject.The appendix also describes algebra categories, including probability, data analysis, equations and inequalities, and numbers and expressions.Source: §A.1 · p. 14, §A.2 · pp. 15–17

A.1 Data Curation

  • Annotators created six problem versions by shifting Essential Information and Reasoned Property between text and diagrams.The versions include Text Dominant, Vision Centric, and Vision Primary; the Text Centric transformation moves diagram information into text.Source: §A.1 · p. 14, Figure 7 · p. 15

A.3 Detailed Statistics of FlowVerse

  • FlowVerse focuses on high-school-level problems from internet sources, excluding advanced college topics such as calculus and graph theory.As DI, EI, and RP are removed across versions, question lengths decrease while answer lengths remain unaffected.Source: Table 9 · p. 15, §A.3 · p. 17
  • Question-length distributions and averages decline as DI, EI, and RP are sequentially omitted from the Text Centric version.Figure 9 plots character-level question-length distributions for Text Centric, Text Limited, Vision Centric, and Vision Dense versions.Source: §A.3 · p. 17

A.4 Details of FlowVerse CoT Evaluation

The appendix details FlowVerse’s prompts and structured chain-of-thought evaluation, then reports error patterns and illustrates how problem representations vary in their text and visual content.

A.4 Details of FlowVerse CoT Evaluation

  • Free-form and multiple-choice questions use distinct prompts, with “first conduct reasoning” included to encourage structured problem-solving.Source: §A.4 · p. 17
  • FlowVerse-CoT-E uses a structured, step-by-step prompt to assess reasoning depth and perceptual accuracy and distinguish reasoning paths.The authors contrast it with evaluations that rely on GPT or human decomposition of responses, which can introduce errors and noise.Source: §A.4 · pp. 17–18
  • Text Plus has the lowest Knowledge and Reasoning Errors and a smaller Visual Perception Error, while vision-heavy versions show higher errors and the largest bubbles.Figure 10 maps problem versions against error types, with bubble radius representing visual-perception-error counts.Source: §A.4 · p. 18, Figure 10 · p. 18

A.5 Qualitative Examples

  • The six problem versions vary the balance of textual and visual information, affecting accessibility and interpretability.The qualitative examples describe Text Centric and Text Limited as text-focused, while Vision Primary conveys almost all information through diagrams.Source: §A.5 · pp. 18–19, Figure 11 · p. 19

B Additional Details of Training Dataset

The appendix describes MathFlow-RP data and training, alongside MathFlow-SFT’s visual and textual annotations for perception and interpretation.

B.1 Details of MathFlow-RP

  • MathFlow-RP targets higher-level abstractions and relationships, using 13,000 annotated instances derived from a 40,000-question solution dataset.Qwen2.5-72B extracts key steps, which are appended sequentially to questions for next-step prediction.Source: §B.1 · p. 20
  • MathFlow⋆GPT-4V demonstrates better performance than GPT-4V on complex mathematical problem understanding and solving in the illustrated comparison.The figure attributes the gains to more effective perception and inference abilities from MathFlow-P-7B.Source: Figure 14 · p. 20

B.2 Details of MathFlow-SFT

  • MathFlow-SFT uses problems and detailed solutions from educational resources, covering plane geometry, algebra, and trigonometry.The dataset is curated for the supervised fine-tuning stage to retain essential visual and textual information for defining solution steps.Source: §B.2 · p. 21
  • Manual MathFlow-SFT annotations label geometric elements, relationships such as congruency or parallelism, and reasoned properties inferred from diagrams and text.Source: §B.2 · p. 21

C.1 More Analysis on MathVision

On MathVision, MathFlow variants are competitive with closed- and open-source baselines, with subject-specific strengths across the two groups.

C.1 More Analysis on MathVision

  • Table 3 compares closed- and open-source MLLMs on MathVision across mathematical subjects, with top scores marked separately for each model group.Source: §C.1 · p. 21
  • MathFlow⋆GPT-V leads among closed-source models in arithmetic, logical reasoning, and geometry, while MathFlow⋆Qwen2-VL-72B performs well in algebra, arithmetic, and topology.Source: §C.1 · p. 21
  • Overall, MathFlow models perform competitively with closed- and open-source baselines on complex mathematical problem-solving.Source: §C.1 · p. 21

C.2 More Analysis of MathFlow Training

The appendix examines MathFlow's training analyses, image-only use, reasoning efficiency, and qualitative geometry examples. It reports a 7B speed–accuracy balance, minor accuracy drops in the image-only setting, and examples where MathFlow avoids errors made by GPT-4V.

C.2 More Analysis of MathFlow Training

  • The training analyses cover six FlowVerse problem versions, perception-model ablations, and distinct procedures for next-step RP prediction and manual EI/RP annotation.Generation hyperparameters were tuned for performance, output stability, and reliability.Source: §C.2 · p. 21, Figure 15 · p. 22, Figure 16 · p. 22
  • Across tested α values, model rankings and relative performance gaps remain qualitatively stable, while the selected setting separates reasoning abilities without over-amplifying noise.Source: §C.2 · p. 23

C.3 More Details of Image-Only Scenario

  • MathFlow extends to the Image-Only variant, created by superimposing each textual question on its diagram, with only minor accuracy degradations.Source: §C.3 · p. 23, Table 17 · p. 24

C.4 Reasoning Efficiency Analysis

  • 44.8 accuracy in 2.51 seconds gives the 7B variant the most favorable speed–accuracy balance among tested Qwen2.5-VL sizes.The 3B model reaches 34.2 accuracy in 0.47 seconds, while the 72B variant reaches 58.3 accuracy in 18.67 seconds; the trained variant also has lower latency than its non-fine-tuned counterpart.Source: §C.4 · p. 23, Table 18 · p. 24

C.5 Qualitative Examples

  • Across three geometry examples, MathFlow solves chord-length, circle-angle, and congruent-triangle problems where GPT-4V makes reasoning or visual-perception errors.Source: §C.5 · p. 24
  • MathFlow's decoupled pipeline produces inspectable intermediate outputs for diagnosing and correcting vision-based failures without retraining the reasoning model.Structured textual representations also leave downstream inference to focus on mathematical reasoning rather than low-level visual decoding.Source: §C.5 · p. 24
Loading 2503.16549v2…