PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartM$^3$: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension

Duo Xu, Hao Cheng, Xin Lin, Zhen Xie, Hao Wang

arXiv:2511.02415v1cs.CV

TL;DR

MLLMs still struggle with chart comprehension, and ChartM3 is introduced to extend chart variety and task complexity. It uses a multi-stage code-driven pipeline to construct visual-reasoning data, with experiments reporting improvements across multiple benchmarks.

  • Problem

    MLLMs still struggle with text-rich images, while ChartM3 is introduced to extend chart variety and task complexity.

    Source: §1 · p. 1

  • Method

    ChartM3 uses RAG-selected chart templates and code-driven stages to generate chart data, images, and Q&A pairs with reasoning, then filters the outputs.

    Source: §1 · p. 2

  • Results

    ChartM3 training significantly improves performance across multiple benchmarks, and CoT-SFT gains at least 12% over the base model on the authors’ benchmarks.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that ChartM3 bridges academic chart-understanding research and practical applications.

    Source: §5 · p. 9

    The work focuses mainly on statistical charts, leaving flowcharts and other visual formats for future extension.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Complex chart understanding tasks demand advanced visual recognition and reasoning capabilities from multimodal large language models (MLLMs). However, current research provides limited coverage of complex chart scenarios and computation-intensive reasoning tasks prevalent in real-world applications. This study proposes an automated multi-stage code-driven pipeline for systematically generating visual reasoning datasets to address these limitations. The pipeline integrates retrieval-augmented generation (RAG) to retrieve professional chart templates and employs chain-of-thought (CoT) strategies to generate reasoning codes that simulate real data distributions, thereby driving chart rendering and question-related statistical computations. Through model-based evaluation, the pipeline enhances chart diversity and data quality. Using this framework, we construct ChartM$^3$, a multi-dimensional and multi-step dataset containing 38K charts and 142K Q&A pairs for training, along with 2,871 high-quality evaluation samples for enabling practical performance assessment. Supervised fine-tuning (SFT) and reinforcement learning (RL) experiments demonstrate that our dataset significantly improves reasoning capabilities and cross-domain generalization performance, enabling smaller models to achieve performance comparable to larger-scale models in complex chart comprehension.

1 Introduction

ChartM3 targets complex chart comprehension with broader chart variety and task complexity. Experiments report improved visual perception, reasoning, and business-analytic performance.

1 Introduction

  • Existing chart datasets emphasize simple chart types, basic perceptual tasks, and Q&A without detailed stepwise reasoning.The paper links these limitations to tensions among data accuracy, complexity, and construction costs.Source: §1 · p. 1
  • ChartM3 extends chart variety and task complexity, and its benchmark targets architectural limitations in complex-chart and cross-chart reasoning.The introduction describes the dataset as reflecting real-world analytics scenarios.Source: §1 · pp. 1–2
  • The pipeline produces 38.4K charts and 142K Q&A pairs, plus a benchmark of 2,871 rigorously verified samples.It uses RAG-selected professional templates, code-driven generation, and model filtering.Source: §1 · p. 2
  • ChartM3 training substantially improves visual perception and reasoning, including business insight extraction and analytical reasoning.The authors frame these gains as helping bridge academic evaluation and real-world applications.Source: §1 · p. 2

2 Related Works

Prior chart datasets have traded off chart and question diversity, scalability, and reasoning support. ChartM3 is positioned as an automated dataset for complex visual-reasoning chart questions.

2 Related Works

  • Early synthetic chart datasets covered few chart types and often used limited topics, templated tasks, and fixed answer formats.PlotQA added real-world data but focused on bar charts, line graphs, and scatter plots.Source: §2 · pp. 2–3
  • Human annotation and real-world chart collection broadened question or chart diversity, but increased costs and limited scalability.ChartQA broadened question forms, while MMC-Instruction, ChartBench, and CharXiv combined real-world charts with manual annotations.Source: §2 · p. 3
  • LLM-based synthesis improved chart-topic richness and Q&A flexibility, yet fine-grained visual-element analysis remained an area for improvement.Examples include ChartLlama, ChartInstruct, and TinyChart.Source: §2 · p. 3
  • Existing datasets still struggle with complex scenario questions and multi-step reasoning, while evaluation pipelines may not objectively measure chart comprehension.The related-work discussion presents ChartM3 as an automated synthesis pipeline for visual-reasoning chart Q&A.Source: §2 · p. 3

3 ChartM3

ChartM3 combines a chart-template database, code-driven data and image generation, multi-step Q&A synthesis, quality checks, and a verifiable-reward approach to chart RL.

3 ChartM3

  • ChartM3 combines RAG-based template selection, multi-stage code-driven generation, model-based verification, and cross-chart comparison tasks.Its workflow spans template construction, chart generation, Q&A generation, data evaluation, and chart RL.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 Template Database Construction

  • The template database organizes 62 scientifically rigorous chart types and includes styled templates aligned with professional and real-world charts.Templates carry attributes such as industry domain, theme, and visualization purpose.Source: §3.1 · p. 4

3.2 Chart Image Generation

  • Generation begins with analytical key questions across 60 domains; RAG selects chart types and templates, while LLM-written code simulates data distributions.Visualization requirements guide code creation and rendering; failed code is iteratively refined using execution errors.Source: §3.2 · pp. 4–5

3.3 Instruction Q&A Generation

  • Q&A generation covers 18 categories across recognition, extraction, calculation, and analysis, using executable analytical code for answers and reasoning paths.Qwen2.5-VL-72B focuses on visual information to reduce code-image misalignment in question generation.Source: §3.3 · p. 5

3.4 Data Evaluation

  • Quality control combines chart-quality classification, multimodal instruction verification, difficulty sampling, and stricter benchmark refinement.Questions referencing non-visualized data and overly simple questions are filtered; benchmark questions also receive manual verification and judge-consistency checks.Source: §3.4 · p. 5

3.5 Chart RL with Verifiable Reward

  • Chart RL uses a hybrid reward combining answer accuracy and output-format compliance, while training uses questions and final answers rather than CoT paths.Multiple-choice and true/false answers use exact matching; fill-in-the-blank and short answers are evaluated by a Qwen3-32B judge.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate ChartM3 as a model-discriminating chart benchmark and test training with its data. SFT and RL improve chart reasoning, while further studies examine dataset composition, CoT annotations, and RL data scale.

4.1 Experimental Setup

  • The evaluation compares proprietary, general open-source, and chart- or OCR-specialized MLLMs using zero-shot inference with consistent default prompts and hyperparameters.Benchmarks include ChartM3-test and established chart, vision, and math evaluations; Qwen-Max serves as the accuracy judge.Source: §4.1 · p. 6

4.2 Experimental Results

  • CoT-SFT gains at least 12% over the base model on ChartM3 benchmarks and improves ReachQA by 7.60% over Qwen2.5-VL-3B and 15.0% over LLaVA-OV-7B.The authors report consistent improvements on perception-oriented and comprehensive out-of-domain benchmarks.Source: Table 3 · p. 7, §4.2 · p. 7
  • ChartM3-test separates models by more than 15 points, including Claude 3.5 Sonnet at 66.18% versus InternVL3-8B at 51.08%.The models score above 86% on ChartQA with minimal differences.Source: §4.2 · p. 7
  • RL improves in-domain ChartM3 by 7.4% and ChartM3-Multi by 5.5%, with a 4.96% gain on CharXiv.The RL model also improves WeMath by 5.14%; MMStar rises from 55.30% to 56.00%.Source: Table 4 · p. 8, §4.2 · p. 8, Table 5 · p. 8
  • SFT performs better on perception-centric tasks, whereas RL gains more on complex reasoning tasks; RL moderately improves basic perception and preserves general capabilities.The authors describe these as complementary strengths, with RL strengthening reasoning patterns rather than introducing new knowledge.Source: §4.2 · p. 8

4.3 Further Study

  • With equal training samples and parameters, ChartM3 yields substantial gains, while ChartGemma changes little and TinyChart degrades performance.The controlled comparison uses Qwen2.5-VL-3B and attributes ChartM3's distinct value to chart diversity, visual complexity, and high-quality CoT annotations.Source: §4.3 · p. 8, Table 6 · p. 8
  • CoT data improves computation-intensive and statistic-related tasks by 8% with the same training-data amount, despite comparable perception-task performance.The ablation compares models trained with and without CoT Q&A.Source: §4.3 · p. 9, Figure 3 · p. 9
  • On ReachQA, performance degrades with 5K samples due to high-variance RL training, while scaling training to 30K samples addresses this instability and yields a 6.95% improvement.Beneficial reasoning patterns are sparser in fill-in-the-blank and short-answer questions, requiring larger datasets to be captured effectively during training.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents a multi-stage, code-driven pipeline for synthesizing chart visual-reasoning Q&A data and applies CoT SFT and RL to the resulting dataset. The reported experiments improve performance across multiple benchmarks, with the framework aimed at connecting chart-understanding research and practical applications.

5 Conclusion

  • The pipeline generates a key question, retrieves chart templates, simulates data distributions with LLM-generated code, renders charts, solves questions, and filters data with models.Its code-driven stages are presented as a way to synthesize visual-reasoning Q&A data.Source: §5 · p. 9
  • ChartM3 is a multi-dimensional, multi-step dataset used for CoT supervised fine-tuning and reinforcement learning.Source: §5 · p. 9
  • Experiments report significant improvements across multiple benchmarks, and the framework is presented as bridging academic chart-understanding research and practical applications.Source: §5 · p. 9

Limitations

The authors identify limits in rendering-language support, chart-format scope, and RL experiment scale.

Limitations

  • The pipeline is primarily Python-based and focuses on statistical charts; RL experiments were not conducted at larger scale.The authors identify other visualization languages, flowcharts, and expanded data and model scales as future directions.Source: Limitations · p. 9

Ethical Consideration

The authors report adherence to the ACL Code of Ethics and licenses for datasets and models, and describe human-annotation consent, privacy, and compensation practices.

Ethical Consideration

  • The authors state that they followed the ACL Code of Ethics and the licenses of open-source datasets and models.Source: Ethical Consideration · p. 9
  • Human annotators gave explicit consent for their annotations to be used exclusively for academic research, and their privacy was to be ensured.Source: Ethical Consideration · p. 10
  • Annotators were compensated according to local standards; data were anonymized and prompts and benchmarks were screened to exclude names, unique identifiers, and offensive content.Source: Ethical Consideration · p. 10

A.1 Data Categories

ChartM3 defines a broad set of chart types, question categories, and visualization domains. Its visual-recognition tasks are divided by difficulty, and the question-category distribution is shown in Figure 5.

A.1 Data Categories

  • The generation pipeline defines 9 major and 62 minor chart types.Table 11 presents the chart types.Source: §A.1 · p. 12
  • It specifies 18 specialized Q&A categories across four dimensions, plus Chart To Markdown.Figure 5 shows the distribution across question subcategories.Source: §A.1 · p. 12, Figure 5 · p. 12
  • Visual Recognition is split into parts A and B because its subcategories have varying difficulty.Source: §A.1 · p. 12
  • The pipeline predefines 60 commonly used visualization domains.Table 13 enumerates the domains.Source: §A.1 · p. 12

ChartM3

Table 7 reports quality-control filtering statistics for ChartM3 components; each ChartM3-Multi data point contains two charts.

ChartM3

  • Table 7 presents filtering statistics for different components of the dataset.Source: Table 7 · p. 12
  • The filtering table covers components of the ChartM3 dataset.Source: Table 7 · p. 12
  • Each ChartM3-Multi data point contains two charts.Source: Table 7 · p. 12

A.2 Dataset Quality Assessment

The appendix reports a 91.24% instruction accuracy after dual verification, describes balanced GRPO sampling and evaluation choices, and illustrates the template database, CoT-SFT comparisons, and generation prompts.

A.2 Dataset Quality Assessment

  • 91.24% instruction accuracy was obtained after evaluating approximately 5,800 samples and modifying or removing 508 cases.Chart quality was also checked with a Qwen2-VL-2B classifier trained on curated examples and validated on 107 instances.Source: §A.2 · p. 12

A.3 GRPO Training Setting

  • GRPO training samples are uniformly drawn from difficulty scores 3–9, with an approximately 1:1 split between rule-based and model-based rewards; the experiments use the k2 KL approximation.Table 9 reports the question-type distribution.Source: §A.3 · pp. 12–13, Table 9 · p. 13

A.4 Explanation for LLM-based Evaluation

  • Traditional ChartQA exact-match and relative-error evaluation can mishandle units and accept year predictions from 1900 to 2100 when the answer is 2000; the paper adds LLM judging and other metrics.The expanded framework includes original and enhanced Relaxed Accuracy for ChartQA and GPT-4o judging for CharXiv.Source: §A.4 · p. 13, Table 10 · p. 14

A.5 Examples of Chart Template Database

A.6 Examples of Evaluation Comparisons

  • Evaluation examples report that CoT-SFT improves complex-chart reasoning, including ranking category medians and correctly referencing renewable-energy data across charts.The multi-chart case contrasts the base model's use of total energy consumption with the CoT-SFT model's answer below 1500 units for 2020 renewable energy.Source: §A.6 · p. 13, Figure 7 · p. 17, Figure 8 · p. 18

A.7 Prompt Templates

  • Prompts divide generation into narrative-and-code data synthesis, two-stage visualization and Q&A construction, and Qwen-Max judging.Visualization prompts use step-by-step requirement analysis before code generation; Q&A prompts synthesize analytical code, then produce reasoning and answers from execution.Source: §A.7 · pp. 13–14, Figure 9 · p. 19
Loading 2511.02415v1…