ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

1 INTRODUCTION

• Existing code-generation benchmarks use text-only inputs, while real-world coding can involve multiple modalities and scientific chart reproduction requires visual understanding and cross-modal reasoning.Researchers may lack the source code or expertise needed to reproduce preferred chart templates.Source: §1 · p. 1
• ChartMimic includes 4,800 curated figure-instruction-code triplets covering 22 chart types and 201 subcategories, with high- and low-level evaluation metrics.Its Direct Mimic and Customized Mimic tasks ask models to generate code for a chart or incorporate new data specified in instructions.Source: §1 · p. 2
• The best open-weight model, InternVL2-Llama3-76B, has an average score gap of 20.6 versus GPT-4o across Direct Mimic and Customized Mimic.The authors report that this gap persists despite open-weight models matching proprietary models on public leaderboards.Source: §1 · p. 2
• ChartMimic is intended to help researchers understand LMM capabilities in visual understanding, code generation, and cross-modal reasoning.The authors envision it as a comprehensive benchmark suite for guiding research on LMM capabilities.Source: §1 · p. 2

2.1 TASK DEFINITION

• Direct Mimic reproduces the supplied chart; Customized Mimic changes its data while retaining the original chart’s aesthetic and design.The two tasks assess visual comprehension and reasoning, and integration of visual and textual information, respectively.Source: §2.1 · pp. 2–3

2 THE CHARTMIMIC BENCHMARK

• ChartMimic evaluates chart-to-code generation by having LMMs produce code from a chart and instruction, then comparing the rendered output with the reference.The benchmark reports both high- and low-level evaluation results.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 DATA CURATION PROCESS

• The curation pipeline filters publication figures and expert selections, adds charts from other platforms, and produces 600 prototypes and 1,200 seed triplets before augmentation to 4,800.The current benchmark focuses on Python code generation; each seed triplet receives three additional augmented triplets.Source: §2.2 · pp. 3–4

2.3 DATA STATISTICS AND DIVISION

• ChartMimic covers 18 regular and 4 advanced chart types across 201 subcategories, with easy, medium, and hard examples and separate test and testmini sets.The test set has 3,600 examples and testmini has 1,200; the average code length is 696.0 tokens (standard deviation 278.4).Source: §2.3 · p. 4, Table 1 · p. 4

2.4 EVALUATION METRICS

• Evaluation averages a GPT-4o high-level similarity score with low-level text, layout, type, and color scores based on traced code execution.The low-level scores use F1; if generated code fails to execute, both high- and low-level scores are set to zero.Source: §2.4 · p. 5

2.5 COMPARISONS WITH EXISTING BENCHMARKS

• Unlike text-only code benchmarks and chart-question benchmarks, ChartMimic uses complex scientific charts and evaluates multimodal chart-to-code generation with multiple metrics.The paper also distinguishes it from similar multimodal benchmarks whose inputs may be simpler or carry data-leakage risks.Source: §2.5 · pp. 5–6, Table 2 · p. 5

3.1 BASELINE SETUP

• The benchmark compares 3 proprietary and 14 open-weight models, ranging from 2.2B to 76.0B parameters, using direct prompting as the baseline setup.The proprietary models are GPT-4o, Claude-3-opus, and GeminiProVision.Source: §3.1 · p. 6

3.2 MAIN RESULTS

• GPT-4o reaches an overall score of 81.2 in Direct Mimic and 83.2 in Customized Mimic; InternVL2-Llama3-76B scores 58.5 and 64.7, respectively.InternVL2-Llama3-76B is the best open-weight model; Table 4 reports the Customized Mimic leaderboard.Source: §3.2 · p. 6, Table 4 · p. 7
• Most open-weight models have execution rates below 75%, and every model except GPT-4o scores below 65.0 in Customized Mimic.The authors identify remaining room to improve visual understanding, code generation, and cross-modal reasoning.Source: §3.2 · p. 7
• Most LMMs improve with user-customized data, but Cogvlm2-llama3-chat-19B and Qwen2-VL-2B decline in overall score and execution rate.The authors attribute typical gains to reduced burden in recognizing chart text and data, while noting that multimodal processing may burden those two models.Source: §3.2 · p. 7

4.1 DIFFERENT COMPLEXITY LEVELS

• GPT-4o’s Direct Mimic score declines from 86.5 (easy) to 77.7 (medium) to 74.8 (hard), with performance falling consistently as complexity rises.For MiniCPM-Llama3-V-2.5, customization raises Easy performance from 46.8 to 59.7 but changes Hard performance from 31.9 to 31.5.Source: §4.1 · pp. 7–8, Figure 3 · p. 7

4.2 DIFFERENT PROMPTING METHODS

• SelfReflection improves GPT-4o and InternVL2-Llama3-76B, but not weaker InternVL2 models; HintEnhanced and Scaffold reduce performance.For GPT-4o, SelfReflection improves Text, Layout, and Type scores but leaves Color unchanged; caption hallucinations and dot-matrix interference help explain the other methods’ declines.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 CORRELATION WITH HUMAN EVALUATION

• Low-level (r = 0.7681) and high-level (r = 0.7041) metrics correlate with human judgments, with the low-level metric showing the higher correlation.The comparison uses human ratings of 1,200 GPT-4o charts generated with four prompting methods on Direct Mimic.Source: Table 6 · p. 9, §4.3 · p. 9

4.4 ERROR ANALYSIS

• GPT-4o’s errors span data dimensions, text, chart type, and color; dimension errors dominate code failures, while missing or extraneous text and nuanced chart semantics remain challenging.GPT-4o can identify similar colors without exactly recognizing the correct ones, and the analysis concludes that it still struggles with code generation, visual understanding, and complex data analysis.Source: §4.4 · p. 9, Figure 4 · p. 9

Large Multimodal Models

• LMMs can perform well on benchmarks yet fall short in practical applications, motivating evaluation grounded in authentic use cases (Liu et al., 2023; Chen et al., 2024; Yue et al., 2023; Lu et al., 2023; Luo et al., 2025; Xie et al., 2024; Koh et al., 2024; Si et al., 2024; Ji et al., 2024).ChartMimic evaluates translating complex visual information into code to assess visual and coding capabilities.Source: §5 · p. 10

Code Generation

• Text-only code-generation benchmarks such as HumanEval, MBPP, APPS, and DS-1000 have limited scope for LMM evaluation (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2023).The passage notes limitations in MMCode’s visual inputs and evaluation, and potential data-leakage risks for Design2Code and Plot2Code; ChartMimic offers 4,800 manually curated data pairs, chart diversity, and fine-grained evaluation.Source: §5 · p. 10

Chart Understanding

• Prior chart question-answering and captioning work assesses chart characteristics or summarizes information in text, while ChartMimic evaluates understanding expressed as code (Masry et al., 2022; Methani et al., 2020; Xu et al., 2023; Wang et al., 2024b; Li et al., 2024c; Zeng et al., 2024; Rahman et al., 2023; Kantharaj et al., 2022).Its fine-grained metrics assess text, chart type, layout, and color.Source: §5 · p. 10

6 CONCLUSIONS

• ChartMimic evaluates chart-to-code proficiency using low- and high-level metrics, but manual curation may introduce bias and its metric does not uniformly score sub-icon details such as markers.Source: §6 · p. 10

B.1 DATA ANNOTATION PRINCIPLES

• Chart diversity matters because visualization practices and requirements vary significantly across fields.Source: §B.1 · p. 17
• Research-paper charts are selected at varying complexity levels because single-format, low-information-density charts are uncommon in practical academic writing.Source: §B.1 · p. 17
• 22.1 for matplotlib gallery code versus 39.8 for Direct Mimic code indicates greater edit distance for Direct Mimic and reduced leakage risk.Source: §B.1 · p. 17
• Dataset construction reflects domain-specific visualization needs, including complex multidimensional data and aesthetic preferences.Source: §B.1 · p. 18

B.2 DATA ANNOTATION PIPELINE

• The pipeline filters arXiv figures to Matplotlib-generated PDFs, then uses expert review and voting to select figures with distinctive visual elements.It narrows approximately 174,100 figures to 15,800 Matplotlib-generated figures, 1,295 after a seven-day review, and a final 279 after independent review and voting.Source: §B.2 · p. 18
• The authors add 600 distinctive charts from other platforms, write Direct Mimic and Customized Mimic triplets, and augment them to produce 4,800 triplets.The 600 prototypes yield 1,200 seed triplets; annotators create three additional augmented triplets per seed by modifying data, color schemes, and mark styles.Source: §B.2 · p. 19

B.3 COMPLEXITY LEVELS

• Chart complexity is assessed through visual components and implementation demands, including code length and structural intricacy, then classified into Easy, Medium, and Hard levels.Visual scoring covers chart type, grouping, markers, text, color, layout, and coordinates, assigning 1–3 points for low, medium, and high complexity.Source: §B.3 · pp. 19–20

B.4 INSTRUCTION EXAMPLES

• The instruction examples show Direct Mimic and Customized Mimic triplets for both a bar chart (bar_28) and a more complex combination chart (CB_29).The examples pair figures, instructions, and code, and show the ground-truth figures rendered from code.Source: §B.4 · p. 20, Figure 7 · p. 21, Figure 8 · p. 22

C CORRELATION BETWEEN TEST SET AND TESTMINI SET

• Minor score differences between Testmini and Test suggest that Testmini effectively mirrors the Test set for Direct Mimic and Customized Mimic evaluations.The passage presents Testmini as a useful evaluation subset for model development, particularly for researchers with limited computing resources.Source: Table 7 · p. 25, Table 8 · p. 25, C CORRELATION BETWEEN TEST SET AND… · p. 25

D CHART TAXONOMY

• The taxonomy comprises 22 chart categories—18 regular types and four advanced types—with regular types further divided into subcategories based on chart features or data characteristics.The advanced types are PIP, Multidiff, Combination, and HR; each advanced-type chart represents a unique subcategory.Source: D CHART TAXONOMY · p. 26
• Taxonomy examples distinguish chart subcategories by attributes such as error-bar orientation and symmetry, radar grouping and fill, pie or ring layering and highlighting, and graph directionality and weight.Error Bar charts have 12 subcategories across orientations, while Error Point charts have five; graph charts distinguish directed or undirected and weighted or unweighted edges.Source: D CHART TAXONOMY · pp. 27–30
• Quiver charts have four subcategories based on vector quantity and data grouping; contour charts have three based on line or fill representation, and histograms have three based on data grouping and positioning.Source: D CHART TAXONOMY · p. 31
• Treemaps have four subcategories based on compactness and edge presence, area charts have two based on markers, and 3D charts have five based on chart form and surface representation.Source: D CHART TAXONOMY · p. 32
• The taxonomy also covers PIP charts with insets, Multidiff charts combining chart types across subplots, Combination charts layering types in one plot, and HR charts that defy or challenge the listed categories.Source: D CHART TAXONOMY · p. 33

E DETAILS OF EVALUATION METRICS

• Evaluation metrics include a GPT-4o score for high-level evaluation and Text, Layout, Type, and Color Scores for low-level evaluation.Source: E DETAILS OF EVALUATION METRICS · p. 35

E.1 GPT-4O SCORE

• GPT-4o compares generated and ground-truth figures across text, layout, type, data, style, and clarity, returning a similarity score from 0 to 100.The figures are supplied to GPT-4o simultaneously.Source: §E.1 · p. 35
• Five Direct Mimic evaluations produced a mean GPT-4o Score of 83.4 with a standard deviation of 0.08.The authors report this as evidence of score stability.Source: §E.1 · p. 36
• A single-round Direct Mimic evaluation costs approximately $5.25, or $2.63 per round using OpenAI batch services.Source: §E.1 · p. 36
• 0.70 for GPT-4o Score versus 0.53 for CLIP Score is the reported correlation coefficient with human evaluation.The authors adopt GPT-4o Score as the high-level evaluation metric.Source: §E.1 · p. 40

E.2 TEXT SCORE

• Text Score is the F1-score of precision and recall for correctly captured text, extracted from generated and ground-truth code by tracing PDF text-drawing calls.The tracer wraps Matplotlib’s draw_text() function.Source: §E.2 · p. 40

E.3 LAYOUT SCORE

• Layout Score is the F1-score of precision and recall over correctly identified subplot layouts recorded from generated and ground-truth code.The layout information records each axis’s position within the figure.Source: §E.3 · p. 41

E.4 TYPE SCORE

• Type Score is the F1-score of precision and recall over chart types identified by tracing plot-function invocations in generated and ground-truth code.Source: §E.4 · p. 41

E.5 COLOR SCORE

• Color Score uses CIEDE2000 color similarity, then computes the F1-score of precision and recall across colors in generated and ground-truth figures.The continuous similarity measure addresses the low similarity scores produced by exact color matching.Source: §E.5 · p. 42

F.1 GENERATION CONFIGURATIONS

• Open-weight models use temperature τ = 0.1, proprietary models use τ = 0 for greedy decoding, and all use a 4096-token generation limit.Open-weight inference uses BF16; all models run on A100 80G GPUs.Source: §F.1 · p. 43

F.2 PROMPTS

• Direct prompting has separate prompts for open-weight and proprietary models, while the appendix also supplies prompts for HintEnhanced, SelfReflection, and Scaffold prompting.The corresponding prompt figures are 38–42.Source: §F.2 · p. 43, Figure 38 · p. 43, Figure 39 · p. 43
• The direct-prompt wording asks for Python code reproducing a STEM-paper picture when no source code is available.Source: §F.2 · p. 43

F.3 CASES OF DIFFERENT PROMPTING METHODS

• The appendix presents HintEnhanced and Scaffold error cases, SelfReflection error and good cases, and directs readers to Appendix I for Direct Prompting errors.The captions identify incorrect figure interpretations in the error cases and a correct reflection in the good case.Source: §F.3 · p. 46, Figure 43 · p. 46, Figure 44 · p. 47

F.4 DETAILS OF MODELS

• The appendix points to Table 9 for the evaluated models’ code/API details; its caption identifies that scope.Source: §F.4 · p. 50, Table 9 · p. 50

G CORRELATION WITH HUMAN EVALUATION

• The human-evaluation set comprises 1,200 figures generated by GPT-4o using four prompting methods on 300 selected Direct Mimic examples.Source: G CORRELATION WITH HUMAN EVALUATION · p. 51
• Three evaluators independently score each figure from 0 to 100, and the final score is their mean.The evaluators are volunteer graduate students with computer-science bachelor’s degrees; the rubric covers chart type, layout, text, data, style, and clarity.Source: G CORRELATION WITH HUMAN EVALUATION · p. 51
• Pearson correlation coefficients are calculated between the automatic metrics and human-evaluation scores.Source: G CORRELATION WITH HUMAN EVALUATION · p. 51

H CORRELATION WITH CHART UNDERSTANDING AND CODE GENERATION

• 0.8714 for rcode exceeds 0.6877 for rchart in the reported correlations with Direct Mimic performance.The comparisons use HumanEval for code generation and ChartQA for chart understanding.Source: H CORRELATION WITH CHART UNDERSTANDING… · p. 52, Figure 48 · p. 52
• The authors state that both chart-understanding and code-generation abilities influence Direct Mimic performance, with code generation having a more significant impact.They identify foundational code-generation abilities as a focus for future multimodal code-generation model development.Source: H CORRELATION WITH CHART UNDERSTANDING… · p. 52

• Cases 1–3 include missing text or types, extraneous types, detail-level text errors, and different colors.Source: Figure 49 · p. 53, Figure 50 · p. 53, Figure 51 · p. 54
• Cases 4–6 include missing text or types, confused types, extraneous text, and different or similar colors.Source: Figure 52 · p. 54, Figure 53 · p. 54, Figure 54 · p. 55
• Figures 55–56 analyze four error types on Direct Mimic, with bracketed numbers indicating error-case counts.Source: Figure 55 · p. 56, Figure 56 · p. 56

J ERROR ANALYSIS OF OPEN-WEIGHT MODELS

• Smaller DeepSeek-VL-7B shows about 53.7% access and parameter errors, 66.7% missing text, and 78.4% missing types, unlike larger models’ GPT-4o-like patterns.Larger models’ code errors predominantly involve dimensional inconsistencies; their text errors mainly involve missing elements, and color recognition identifies similar or identical chromatic properties.Source: J ERROR ANALYSIS OF OPEN-WEIGHT MODELS · p. 56, Figure 57 · p. 57

K ETHICS, SOCIETAL IMPACT AND SCALABILITY OF CHARTMIMIC

• The dataset uses scientific figures from arXiv papers under CC BY 4.0, and manual code annotation applies controls against biased content and factual misrepresentation.Source: K ETHICS, SOCIETAL IMPACT AND… · p. 57
• ChartMimic’s scientific-data benchmark is presented as supporting LMM development for complex visualizations and benefiting scientific communication, data analysis, and knowledge dissemination.Source: K ETHICS, SOCIETAL IMPACT AND… · p. 57
• Its modular design supports adding chart types and metrics; arXiv’s update cycle supports dataset expansion, while evaluation metrics can extend to other languages and evolving human preferences.Source: K ETHICS, SOCIETAL IMPACT AND… · p. 57