Source-linked AI summary
ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Zhiyuan Liu, Maosong Sun
TL;DR
Existing MLLMs lose dense chart information through textual descriptions and face poor executability and limited training data in chart-to-code tasks. ChartCoder combines a Code LLM backbone with the Chart2Code-160k dataset and Snippet-of-Thought generation, outperforming existing open-source MLLMs on chart-to-code benchmarks. The study is limited to a 7B-parameter model and leaves larger models and more diverse charts for future work.
Problem
Existing MLLMs often lose dense chart information through natural-language descriptions, while chart-to-code models face limited executability and insufficient large-scale, diverse training data.
Method
ChartCoder uses a Code LLM language backbone, Chart2Code-160k with 160k pairs across 27 chart types, and Snippet-of-Thought step-by-step generation.
Results
ChartCoder, with 7B parameters, outperforms existing open-source MLLMs on chart-to-code benchmarks and achieves the best performance among open-source MLLMs across chart-to-code tasks.
Takeaways & Limitations
The paper presents a dedicated chart-to-code MLLM, dataset, and step-by-step generation method for preserving chart information in executable code.
Takeaways & Limitations
ChartCoder was trained only with 7B parameters because of computational-resource constraints; larger models and more complex, diverse charts remain future directions.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding tasks. However, interpreting charts with textual descriptions often leads to information loss, as it fails to fully capture the dense information embedded in charts. In contrast, parsing charts into code provides lossless representations that can effectively contain all critical details. Although existing open-source MLLMs have achieved success in chart understanding tasks, they still face two major challenges when applied to chart-to-code tasks: (1) Low executability and poor restoration of chart details in the generated code and (2) Lack of large-scale and diverse training data. To address these challenges, we propose \textbf{ChartCoder}, the first dedicated chart-to-code MLLM, which leverages Code LLMs as the language backbone to enhance the executability of the generated code. Furthermore, we introduce \textbf{Chart2Code-160k}, the first large-scale and diverse dataset for chart-to-code generation, and propose the \textbf{Snippet-of-Thought (SoT)} method, which transforms direct chart-to-code generation data into step-by-step generation. Experiments demonstrate that ChartCoder, with only 7B parameters, surpasses existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code excitability. Our code is available at https://github.com/thunlp/ChartCoder.
1 Introduction
Existing MLLMs often describe charts in natural language, which overlooks dense chart information and performs poorly on executable chart-to-code generation. ChartCoder addresses these challenges with a Code LLM backbone, Chart2Code-160k, and Snippet-of-Thought generation.
- Natural-language chart descriptions overlook dense information, whereas code provides a lossless representation of chart details.
- Existing open-source MLLMs show poor chart-to-code performance and limited execution rates because they are not well-aligned with code generation.
- ChartCoder is a dedicated chart-to-code MLLM using a Code LLM as its language backbone and a two-stage training paradigm.
- With only 7B parameters, ChartCoder outperforms existing open-source MLLMs on chart-to-code benchmarks and generates more correct, executable code.
- 160k chart-code pairs across 27 chart types form the Chart2Code-160k dataset for addressing chart-to-code data scarcity.
- Snippet-of-Thought transforms direct chart-to-code generation into step-by-step generation to emphasize critical information and strengthen reasoning.
2 Related Works
Chart understanding spans low- and high-level tasks, while recent MLLMs address chart-related problems end to end. Chart-to-code generation extends multimodal code generation by requiring faithful reproduction of complex charts.
- Chart understanding includes both low-level and high-level tasks, with prior pipeline methods limited by error accumulation across stages.
- Recent chart-domain MLLMs trained on extensive chart-specific datasets achieve strong performance across chart-related tasks.
- Multimodal code-generation benchmarks evaluate MLLMs on visual code problems, including HTML web-page creation and visual reasoning.
- Chart-to-code generation requires code that accurately reproduces a given chart, including its text and data, and dedicated research had been absent before this work.
3 Chart2Code-160k Dataset
Chart2Code-160k addresses the scarcity and structural demands of chart-to-code data through large-scale synthetic generation and step-by-step code decomposition. It contains diverse chart structures, executable code, and SoT examples designed to preserve critical details.
- Direct Chart-to-code Generation: Chart-to-code datasets require one-to-one chart-code mappings, diverse chart types and structures, and syntactically executable outputs.These requirements make chart-to-code data more demanding than chart-reasoning datasets.
- Direct Chart-to-code Generation: Chart2Code-160k generates code first and executes it to produce corresponding chart images, enabling large-scale chart-code pair construction.The generation process uses LLM-produced domain keywords, simulated data, and chart templates.
- Direct Chart-to-code Generation: The dataset covers 27 chart types using 79 manually written template codes, supplemented with functions and parameters to increase structural diversity.Examples include plt.text() and hatch='/' for encouraging varied chart functions and visual parameters.
- Step-by-step Chart-to-code Generation: SoT converts direct chart-to-code generation into four stages: chart layout, data and colors, critical details, and complete code.Each stage combines textual explanations with code snippets to emphasize key information.
- Step-by-step Chart-to-code Generation: To reduce inconsistencies from step-by-step hallucinations, 50k pairs are decomposed from the 160k dataset using placeholders or default parameters.The decomposition preserves consistency between intermediate snippets and the final executable code.
- Dataset Analysis: Chart2Code-160k contains 160k instruction-tuning pairs spanning 27 chart types, with syntactically correct and executable code aligned to chart structures.Manual and GPT-4o-based evaluations were used to assess generated chart quality.
4 ChartCoder Model
ChartCoder combines a vision encoder, connector, and Code LLM backbone in a two-stage training process. Its design targets chart structure perception and chart-to-code generation using high-resolution inputs and dedicated data.
- Model Architecture: ChartCoder uses a pretrained SigLIP-384 vision encoder, a two-layer MLP connector, and the DeepSeek Coder 6.7B language backbone.The encoder extracts visual features, the connector projects them into word-embedding space, and the backbone generates responses.
- Model Architecture: Any Resolution resizes and patchifies chart images so ChartCoder can process high-resolution visual details.This addresses the resolution limits of standard vision transformers such as CLIP and SigLIP.
- Model Training: ChartCoder follows two training stages: chart-to-text alignment followed by chart-to-code instruction tuning.The stages are designed to support chart structure perception and code generation.
- Model Training: During alignment, the vision-language connector is pretrained while the language and vision encoders remain frozen.Alignment uses chart caption, chart-to-table, and related multimodal corpora.
- Model Training: During instruction tuning, all three modules are jointly fine-tuned with Chart2Code-160k and additional code-related data.Additional data include ChartQA PoT and ChartLlama chart-to-chart examples.
- Evaluation: Table 3 evaluates baseline models across ChartMimic, Plot2Code, and ChartX, while Table 4 reports detailed ChartMimic high-level subscores.The supplied passages specify the evaluation layouts but do not provide the complete table values.
5 Experiments
ChartCoder is evaluated against general-domain, proprietary, and chart-domain MLLMs across three chart-to-code benchmarks. It achieves strong restoration and execution-related results, while ablations examine its code backbone, resolution, dataset, and use of code for chart understanding.
- Baselines and Benchmarks: ChartCoder is compared with general-domain, proprietary, and chart-domain MLLMs on ChartMimic, Plot2Code, and ChartX.The evaluation includes open-source models at multiple scales and revises Plot2Code’s rating calculation to include all results.
- Main Results: Three of four low-level ChartMimic subscores for ChartCoder exceed GPT-4o.The low-level evaluation compares code similarity in text, layout, type, and color.
- Main Results: ChartCoder achieves the best performance among open-source MLLMs across all chart-to-code tasks and exceeds some proprietary models.On ChartMimic, it surpasses MiniCPM-Llama3-V2.5 and InternVL2-8B, which report average scores of 26.7 and 34.6, respectively.
- Ablation Study: Using a Code LLM backbone significantly improves execution rate, low-level scores, and high-level scores over a general LLM backbone.The Code LLM also reduces syntax and type errors in non-executable code.
- Ablation Study: Removing 50k step-by-step examples notably lowers low-level and high-level scores, especially for text content and data.Directly fine-tuning Qwen2-VL-7B on Chart2Code-160k significantly improves all metrics.
- Ablation Study: Adding code to chart images helps MLLMs understand chart details, especially chart types and data.The comparison uses Image-only and Image-with-Code inputs on the MMC True/False benchmark.
6 Conclusion
The work addresses chart-to-code generation with ChartCoder, Chart2Code-160k, and Snippet-of-Thought data. Experiments show that ChartCoder outperforms existing open-source MLLMs.
- 6 Conclusion: ChartCoder uses a Code LLM backbone, Chart2Code-160k supplies large-scale chart-to-code data, and SoT generates step-by-step training data.The paper presents these as its three main responses to chart-to-code challenges.
- 6 Conclusion: Experiments show that ChartCoder outperforms existing open-source MLLMs.
Limitation
The study’s main limitation is that computational constraints restricted ChartCoder training to 7B parameters. Future work may examine larger models, more diverse charts and code, and other image types.
- Limitation: Computational constraints limited ChartCoder training to 7B parameters.The authors state that larger models could potentially achieve better performance.
- Limitation: Future work may explore more complex and diverse charts and codes, as well as image types such as HTML.
Ethical Statement
The research uses publicly available models and datasets with citations, aiming to reduce the risk of toxic content.
- Ethical Statement: The study uses publicly available models and datasets with proper citations.The authors characterize these resources and prompts as widely used and non-toxic.
- Ethical Statement: The authors state that this approach minimizes the risk of generating toxic content.
A.1 Implementation Details
ChartCoder's data generation uses GPT-4o for both direct and step-by-step chart-to-code processes. Training combines vision-language alignment with full-model instruction tuning.
- GPT-4o-2024-08-06 generates data through both direct and step-by-step processes.
- ChartCoder initializes SigLIP-384 as the vision encoder and DeepSeek Coder 6.7B as the language model.
- Training has alignment and instruction-tuning stages, with only the connector trained during alignment and the entire model trained during instruction tuning.Alignment uses chart-to-text data; instruction tuning runs for 1 epoch with batchsize 128.
A.2 Benchmark Details
The evaluation covers chart redrawing and chart-to-code benchmarks, with additional studies examining architecture, readability, data composition, chart quality, and generated outputs. Benchmark-specific metrics and qualitative comparisons are reported across these analyses.
- Benchmark Details: ChartMimic evaluates chart redrawing on ArXiv figures, emphasizing original style and appearance preservation.The Direct Mimic task on testmini reports execution success rates alongside low-level and high-level scores.
- Benchmark Details: Plot2Code evaluates code generation from Matplotlib galleries using Pass Rate, Text-Match, and Rating.The evaluation uses its Direct Asking task.
- Benchmark Details: ChartX evaluation uses the Redrawing task and reports GPT score.
- Ablation Studies: A language-backbone ablation compares Qwen2.5-7B and Qwen2.5 Coder-7B with other backbone choices.The reported results favor Code LLMs over general LLMs, while Qwen2.5 Coder does not perform as well as DeepSeek Coder.
- Readability Evaluation: The readability study evaluates Naming Conventions, Code Structure, Comments, and Logical Clarity using a total score of 100.It compares ChartCoder outputs with human-annotated ground-truth code from ChartMimic.
- Qualitative Analyses: Figures provide qualitative comparisons of error types, image versus image+code inputs, and bar-chart code generated by different models.Figures 7 and 8 show prompts used for dataset quality evaluation.
- Dataset and Chart Quality: Chart2Code-160k analysis reports chart-type proportions and evaluates generated-chart quality against real-world charts.GPT-4o-2024-08-06 performs the quality evaluation, whose prompts appear in Figure 8.