PaperlayerView exact revision on arXiv

Source-linked AI summary

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang, Yifei Tao, Dongxing Mao, Yang Wan, Jingru Tan, Min Zeng, Min Li, Alex Jinpeng Wang

arXiv:2510.17932v4cs.SEcs.AI

TL;DR

Existing chart-to-code benchmarks do not cover the full range of realistic interactive workflows, including complex edits and charting long, raw tables. Chart2Code introduces a three-level benchmark and evaluates executable code alongside rendered-chart fidelity. Across the tasks, model performance degrades substantially on complex editing and long-context visualization.

  • Problem

    Existing benchmarks do not systematically cover realistic interactive chart-coding scenarios such as editing figures or turning long, raw tables into plots.

    Source: §1 · p. 1

  • Method

    Chart2Code organizes chart generation into reproduction, editing, and long-table-to-chart tasks, with evaluation of both code executability and visual fidelity.

    Source: §1 · p. 2

  • Results

    Current models perform reasonably well on simple reproduction but degrade substantially on complex editing and long-context visualization tasks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings point to persistent chart-to-code challenges and a need for future models with stronger reasoning, generalization, and robustness in real-world visualization scenarios.

    Source: §6 · p. 9

    All benchmark tasks are in English, and LLM-based evaluation may introduce biases and inaccuracies, especially for subtle visual details.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.

1 Introduction

Chart2Code addresses the mismatch between existing chart-to-code benchmarks and realistic interactive workflows with a hierarchical benchmark spanning reproduction, editing, and long-table chart generation. Its evaluation covers both executable code and visual fidelity, and results show that model performance drops on the more complex tasks.

1 Introduction

  • Existing benchmarks face saturation and do not systematically cover realistic workflows such as figure editing and charting long, raw tables.The paper cites GPT-4o's 82.2% score on ChartMimic as an example of saturation.Source: §1 · p. 1
  • Chart2Code introduces three progressively challenging levels: reference-chart reproduction, complex chart editing, and long-table-to-chart generation.The levels target increasingly demanding real-world charting scenarios.Source: §1 · p. 2, Figure 1 · p. 2
  • LMM performance is promising on simple reproduction but deteriorates sharply on complex editing and long-context data-to-chart generation.The authors identify these tasks as unsolved challenges for reliable visualization assistants.Source: §1 · p. 2
  • The benchmark's evaluation protocols jointly assess code executability and the visual fidelity of rendered charts.The paper also reports an empirical study of mainstream LMMs.Source: §1 · p. 2

2 Related Work

Prior chart research spans question answering, chart-to-code generation, editing, and other visual-generation tasks. The paper positions Chart2Code around practical chart-to-code workflows, particularly scenarios not directly addressed by those neighboring benchmarks.

Agentic Benchmarks

  • Chart-to-code sits alongside benchmarks for GUI automation, agentic coding, tool use, research assistance, and chart reasoning.The paper frames chart-to-code as a practical task in everyday workflows for researchers and professionals.Source: §2 · p. 3

Chart Understanding to Code Generation

  • ChartQA and ChartXiv address chart question answering, with ChartXiv focusing on scientific charts and expert-designed questions.ChartXiv exposes a gap between multimodal models and human performance.Source: §2 · p. 3
  • ChartMimic formalizes code synthesis from chart images and instructions, while ChartEdit and ChartM3 emphasize interactive code modification.Other works cover vector graphics, figure editing, VQA, document understanding, and infographic generation, but do not directly address this paper's chart-to-code formulation.Source: §2 · p. 3
  • Despite high ChartMimic and ChartEdit scores, GPT-4o still struggles with realistic chart-to-code tasks.The authors use this gap to motivate a more challenging benchmark for reliable evaluation.Source: §2 · p. 3

3 Chart2Code: From Visual Charts to Code

Chart2Code formalizes chart-to-code generation and organizes it into progressively harder tasks, from reproducing and editing charts to generating charts from long tables. The benchmark combines executable-code and visual-fidelity evaluation.

3.1 Task Definition of Chart2Code

  • Chart2Code formulates generation as executable Python code conditioned on a reference chart, a natural-language instruction, and optional underlying data.The optional data source can be raw text, a visual table, or a structured file.Source: §3.1 · p. 3

3.2 Hierarchical task design

  • The hierarchy progresses from visual-only reproduction or style transfer with data, through chart edits and analysis, to long-table chart generation requiring long-context reasoning, retrieval, calculation, and code generation.Level 2 includes operations such as chart-type changes, trend lines, and data-analysis tables.Source: §3.2 · p. 4

3.3 Data Statistics and Analysis

  • The benchmark contains 2,186 tasks across three levels, with 863, 1,010, and 313 tasks for Levels 1, 2, and 3, respectively.Its chart-family coverage is 22, 19, and 12 for the three levels; Level 3 uses 245 charts and 313 instructions derived from web-sourced long tables.Source: §3.3 · p. 4

3.4.1 Overall

  • Evaluation uses executable rate as a primary metric and assesses similarity at both code level and chart level.Code-level frameworks include rule-based and LLM-based evaluation, while chart-level evaluation uses LMMs to assess visual fidelity.Source: §3.4.1 · p. 4

3.4.2 Code-Level

  • The code-level Base method parses plotting objects and evaluates eight visual-fidelity dimensions; an LLM-based metric separately scores code-level similarity.The authors describe Base as broader in dimensional coverage and more applicable, faster, and more accurate than a four-dimension heuristic.Source: §3.4.2 · pp. 4–5

3.4.3 Chart-Level

  • Chart-level visual fidelity is scored by GPT-5-mini through multidimensional similarity scoring, and a 20-student human study found strong correlation with LMM scores.Participants assessed ground-truth and generated charts across multiple dimensions.Source: §3.4.3 · p. 5

4 Experiments

Across chart reproduction, editing, and long-table generation, model code can remain executable or structurally plausible while rendered-chart fidelity lags; performance also falls as task complexity rises. The experiments compare 29 models and analyze difficulty scaling, evaluation metrics, and failure modes.

4.1 Experiments Setup

  • The experiments test 29 open-source and proprietary models; evaluations use native-resolution images and model-specific maximum input-pixel settings.Models in the 7B–9B range use two NVIDIA V100 GPUs with model parallelism.Source: §4.1 · pp. 5–6

4.2.1 Level-wise Comparison of Models

  • Across Levels 1–3, proprietary models generally fare better on simpler tasks, but editing and long-table generation sharply reduce performance and visual fidelity.Level 3 overall execution rates remain below 50%, primarily because long-context inputs exceed maximum input limits.Source: §4.2.1 · p. 6, Table 3 · p. 7, Table 4 · p. 8

4.2.2 Analysis

  • Across levels, Base-Score and LLM-Score track each other more closely than either tracks LMM-Score, exposing a gap between code assessment and visual similarity.The analysis reports that rule-based and LLM-based code-only assessments correlate poorly with actual visual similarity.Source: §4.2.2 · p. 7, Table 3 · p. 7, Table 4 · p. 8
  • Level 2 and Level 3 results show that strong code-level scores do not ensure visual fidelity, while execution success declines steeply with task complexity.For Level 3, long-context inputs exceed model limits and chart-level fidelity is the main bottleneck.Source: §4.2.2 · pp. 6, 8, Table 4 · p. 8, Table 5 · p. 8
  • Larger open-source models narrow the Level 2 execution gap, but chart-level fidelity gains remain modest.Qwen2.5-VL-72B reaches 74.55 ER with chart-level score 16.69; InternVL3-38B reaches 69.80 ER with chart-level score 13.32.Source: §4.2.2 · p. 7, Table 4 · p. 8
  • On Level 2, thinking variants improve some procedural measures, but their chart-level fidelity gains are inconsistent.MiMo-VL-7B-RL ER rises from 17.23 to 26.14, while MiMo-VL-7B-SFT chart-level score rises from 15.23 to 16.38.Source: §4.2.2 · p. 7, Table 4 · p. 8
  • Failure modes shift from perception at Level 1 to reasoning at Level 2 and long-context understanding at Level 3.Reported errors include data-extraction inaccuracies, visual-style mismatches, layout inconsistencies, and axis-related issues.Source: §4.2.2 · p. 8

4.3 Discussion.

  • Difficulty analyses show performance declining from easier to harder cases, with almost all models producing low-fidelity charts on Level 3.Figure 3 also contrasts reasonable Level 1-to-Level 2 generalization on LLM-Score with lower LMM-Scores, especially for open-source models.Source: §4.3 · p. 9, Figure 3 · p. 9

5 Limitations

Chart2Code is limited to English tasks and uses LLM-based judges to assess code correctness and visual fidelity, which may introduce biases and inaccuracies.

5 Limitations

  • All Chart2Code tasks are currently in English, leaving multilingual chart-to-code evaluation outside the benchmark's scope.The authors identify multilingual expansion as a direction for future work.Source: §5 · p. 9
  • LLM-based judges enable scalable evaluation but may produce biases and inaccuracies relative to human assessment, especially for subtle visual details.The authors identify more robust evaluation protocols as a future direction.Source: §5 · p. 9
  • The authors identify multilingual expansion and more robust evaluation protocols as future work to improve coverage and reliability.Source: §5 · p. 9

6 Conclusion

Chart2Code covers three progressively challenging chart-to-code levels, and its evaluation of 29 LMMs finds substantial performance declines on complex editing and long-context visualization tasks. The paper also reports its use of LLMs for editing and experiments, but not for core research ideas or conclusions.

6 Conclusion

  • Chart2Code organizes chart reproduction, chart editing, and long-table-to-chart generation into three progressively challenging levels.Source: §6 · p. 9
  • Across 29 state-of-the-art LMMs, performance degrades substantially from simple reproduction to complex editing and long-context visualization tasks.The conclusion describes these as persistent, unsolved challenges in chart-to-code generation.Source: §6 · p. 9

A LLM Usage Statement

  • The authors used LLMs for grammatical and stylistic editing, experiments, and evaluation, but not for core research ideas, methods, designs, or conclusions.Source: A LLM Usage Statement · p. 14

B User-Centric Case Studies

The case studies examine user-facing chart tasks that combine data interpretation, instruction following, and visual rendering. They show recurring errors in editing and basic perception, including failures on tasks described as routine.

B User-Centric Case Studies

  • A Level 2 inventory-editing task requires models to group categories by inventory sufficiency, encode the groups visually, and preserve the reference style.This combines reasoning over data values, instruction constraints, and visual design.Source: B User-Centric Case Studies · p. 14
  • Observed failures include incorrect data analysis, mismatched visual styles, and missing or misconfigured chart components.The errors indicate difficulty maintaining consistency between data, instructions, and visual representation.Source: B User-Centric Case Studies · p. 14
  • Even seemingly routine tasks can fail, while the correct example requires alignment between data extraction, reasoning, and visual rendering.Source: B User-Centric Case Studies · p. 14
  • Models also make errors on basic chart-perception tasks, suggesting weak visual grounding and fine-grained reasoning.The authors caution that unreliable performance in user-level scenarios is unlikely to transfer to more complex long-context and compositional tasks.Source: B User-Centric Case Studies · p. 14

C Fine-Grained Visual Perception Analysis

Controlled examples reveal inconsistent fine-grained visual perception, while the data-curation pipeline collects and filters chart images and raw data before annotating the benchmark's three task levels.

C Fine-Grained Visual Perception Analysis

  • Even when examples isolate visual attributes, models inconsistently identify title structure, legend position, grid-line style, tick-label rotation, and axis orientation.The authors connect these weaknesses in precise spatial and stylistic details to the need for reliable low-level attribute extraction in chart-to-code generation.Source: C Fine-Grained Visual Perception… · p. 14

D Data Curation

  • Chart and corresponding raw data were collected from multiple sources to construct a comprehensive, challenging benchmark.Source: D Data Curation · p. 21

D.1.1 Data curation

  • Chart sources include roughly 5,000 recent arXiv paper charts, 1,000 library or tutorial examples, and 150 difficult ChartMimic charts.Raw data comes from sources including Kaggle, annual reports, and company websites, with varied formats and table lengths.Source: §D.1.1 · p. 21

D.1.2 Data filtering

  • Chart selection uses initial filtering followed by expert assessment of data, visual, and programming complexity; raw data is screened for plotting suitability and diversity.The chart-selection process yields 719 unique reference figures, while the raw-data selection includes 71 Excel files, 108 figures, and 36 text files.Source: §D.1.2 · p. 21

D.1.3 Data Annotation

  • Level 1 annotations recreate charts, Level 2 annotations produce 1,010 edited entries, and Level 3 annotations produce 313 long-table entries.Level 3 annotation includes formulating statistical requirements and using preprocessing scripts to clean and transform tables.Source: §D.1.3 · p. 21

D.2 Chart Image Data

  • The chart-image collection draws on academic papers, visualization tutorials and communities, and existing datasets, then retains charts selected for complexity and expert agreement.Five experts score data, visual, and programming complexity; charts with disagreements receive a final decision from two core researchers.Source: §D.2 · p. 22

D.3 Raw Data Filtering

Raw Data Filtering combines automated table selection with manual quality and diversity checks. The annotation process used interactive guidance, consensus resolution, and stated participant protections.

D.3 Raw Data Filtering

  • Scripts selected numerical, time-series, or categorical tables, then manual review removed incomplete or poorly formatted data while preserving domain diversity.The resulting database contains 71 Excel files, 108 structured data figures, and 36 semi-structured text files.Source: §D.3 · pp. 22–23

D.4 Human Annotation Details and Ethical Considerations

  • Annotators received interactive training and real-time feedback, with ambiguous cases resolved through consensus discussions among annotators and authors.The task involved undergraduate students recruited for coding expertise; some annotators were authors and received no monetary compensation.Source: §D.4 · p. 23
  • Participants were informed about the study and data use; the authors report anonymity and a right to withdraw, and did not seek IRB approval.They cite standard data processing, no intervention in participants’ personal lives, and use of non-sensitive public data as the reasons.Source: §D.4 · p. 23

E.1 Details Evaluation results

The evaluation analyses show that code-level scores can diverge substantially from rendered-chart fidelity, even when rule-based and LLM-based code assessments broadly agree. The comparisons also show that code-level metrics can reflect some visual-fidelity trends without matching their actual scores.

E.1 Details Evaluation results

  • Gemini-3-pro scores 94.44% Type on Base and 73.75% Type on LLM, while its Chart-Level LMM score is 40.72%.The authors describe this as a substantial code-to-chart disparity; Fig. 5 compares Base-Score and LLM-Score across code-level evaluation dimensions.Source: §E.1 · p. 28, Figure 5 · p. 29
  • Code-Level metrics evaluate script syntax and logic but miss rendering artifacts including occlusion, poor scaling, and color clashing; Gemini-3-pro scores 94.44% Base Type and 73.75% LLM Type, versus 40.72% Chart-Level LMM.Source: §E.1 · p. 28
  • Base-Score and LLM-Score show noticeable divergence from LMM-Score, which assesses rendered, pixel-level visual fidelity.Fig. 6 shows that code-level metrics can still reflect the overall trend of visual fidelity to some extent.Source: §E.1 · p. 28, Figure 6 · p. 29
  • Across ten representative cases, proprietary models often score at least 0.2 on LLM evaluation but typically no more than 0.35 on LMM evaluation.The authors report lower LMM scores for open-source models across the tasks and say these assessments align more closely with human visual perception.Source: §E.1 · pp. 28, 31

E.2 Correlation Between Code Generation Quality and Visual Fidelity

The analysis finds positive associations between code-generation quality and rendered-chart fidelity, while emphasizing that code scores do not fully capture visual outcomes. It also reports that LLM-based scoring is more sensitive than rule-based scoring to some visual defects, and that reasoning-related structural gains coincide with higher visual scores.

E.2 Correlation Between Code Generation Quality and Visual Fidelity

  • Gemini-3-pro and GPT-5.2 have the benchmark’s highest LMM-Scores, 45.42 and 43.73, respectively.The authors report that models with structurally and semantically correct code consistently achieved higher visual-fidelity scores.Source: §E.2 · p. 31
  • LLaVA models score below 3.0 on LMM and have low scores across code dimensions, while Gemini-3-pro and GPT-5.2 lead code-level evaluations.The authors characterize code-level quality as a strong predictor of visual fidelity, while concluding that a code-visual gap remains, particularly for open-source models.Source: §E.2 · p. 31
  • LLM-Score is more sensitive than Base-Score to visual defects; Claude-Sonnet-4 scores 85.66 on Base Text, 70.69 on LLM Text, and 32.36 on LMM.The authors say LLM evaluation can penalize syntactically correct but visually suboptimal code, including overlapping text or misaligned labels.Source: §E.2 · p. 31
  • MiMo-VL-7B-RL’s Grid Base-Score rises from 61.70 to 74.48, while its LMM-Score rises from 21.43 to 30.04 versus the non-thinking variant.The authors connect the code-structure improvement to reasoning about coordinate systems and layouts.Source: §E.2 · p. 31

E.3 Evaluation Case

The case studies compare rule-based, LLM-based, and LMM-based evaluations of the same chart instances. They report broad agreement on coarse model ranking alongside complementary assessments of code structure and rendered visual fidelity.

E.3 Evaluation Case

  • The framework uses Base for code-level structural and parameter consistency, LLM for semantic assessment of generated code, and LMM for visual similarity of rendered charts.For each case, it reports final and subdimension scores with explanations.Source: §E.3 · pp. 31–32
  • Base and LLM show relatively consistent scoring trends, but both can diverge from LMM’s pixel-level visual-fidelity assessment.The authors caution that code correctness, even when semantically assessed, does not necessarily yield high visual similarity.Source: §E.3 · p. 32
  • Across cases, the three methods show some correlation in coarse-grained model ranking, but differ in what they detect.Base and LLM are more sensitive to structural and logical correctness; LMM better captures layout shifts, axis misalignment, and styling differences.Source: §E.3 · p. 32
  • The case studies conclude that considering code-level correctness together with chart-level visual fidelity gives a more comprehensive and interpretable assessment.The authors state that relying solely on rule-based or code-based evaluation may lead to incomplete conclusions.Source: §E.3 · p. 32

E.4 Error Case Study:

The error study traces failures across all three levels to shared perception and reconstruction problems, with the dominant bottleneck shifting from perception and extraction to reasoning and long-context processing.

Cross-level error types

  • Across levels, recurring errors include color consistency, component positioning, data extraction and encoding, and axis scaling.These errors reflect limits in visual perception and chart-code reconstruction.Source: §E.4 · p. 32

Level 1

  • At Level 1, weak visual perception and data extraction cause missing or inaccurate details even when generated code is executable.Grid configurations, marker styles, line patterns, and plotted values are common points of failure.Source: §E.4 · p. 32

Level 2

  • At Level 2, complex compositional edits expose reasoning limits that prevent models from satisfying instructions while preserving the reference chart’s context.This reasoning bottleneck explains the consistent performance gap between Level 2 and Level 1.Source: §E.4 · p. 32

Level 3

  • Level 3 degradation stems mainly from long-context limits that weaken data extraction and instruction comprehension, reducing execution success and visual fidelity.Extensive tabular data consumes context and can trigger the “lost-in-the-middle” phenomenon.Source: §E.4 · p. 32

F Limitations

The stated limitations are English-only tasks and LMM-based judging; qualitative cases also illustrate failures in complex edits and long-table processing.

Linguistic diversity and generalization

  • English-only annotations and prompts leave multilingual generalization and text rendering in non-English contexts outside the benchmark’s tested scope.The paper specifically notes that it does not fully evaluate non-English queries or multi-byte text such as CJK characters.Source: F Limitations · pp. 32, 54

Qualitative error cases

  • Qualitative examples show that misplaced legends can obscure titles and incorrect data distributions can undermine faithful chart reproduction.The cases attribute these mismatches to insufficient data accuracy and recognition.Source: F Limitations · p. 39

Complex editing cases

  • Composite edits requiring shared axes and conditional annotations can overwhelm reasoning and scheduling, collapsing layout constraints and omitting analytical elements.One case omits a maximum-error highlight.Source: F Limitations · p. 51

Long-context cases

  • Long-table cases illustrate context-related failures: heatmap data may be incomplete, while very large inputs can cause truncation, omissions, and instruction forgetting.One example also reports semantic hallucinations under context overload.Source: F Limitations · pp. 52–53

Evaluation bias in LMM-based judges

  • LMM-based judging enables scalable evaluation but may favor familiar code patterns or miss subtle visual artifacts, leaving more objective visual metrics an open challenge.The authors identify potential bias and inaccuracies relative to human assessment.Source: F Limitations · p. 54

G.1 Correlation between Automated Metrics and Human Evaluation

Human ratings generally track LMM chart-quality scores, especially in relative rankings, though score calibration and CRD linear correlation remain imperfect. Across the evaluation framework, visual fidelity lags code-level performance, which is assessed through complementary code- and image-based measures.

G.1 Correlation between Automated Metrics and Human Evaluation

  • LMM scores correlate strongly with human ratings on most tasks, though CRD linear correlation is weaker; rank agreement is stronger despite score discrepancies and upward bias.The scores are more reliable for ranking chart quality than for matching human scores precisely, and tend to overestimate quality.Source: §G.1 · p. 54

G.2 Performance Comparison

  • High execution and code-level scores do not ensure visual fidelity, which remains a difficult frontier and often declines relative to code-level scores.The reported gap concerns fine-grained details such as colors, marker styles, and layout constraints.Source: §G.2 · pp. 54, 56

H.1 Overall

  • The overall evaluation first measures code executability, then uses code-based LLM assessment and image-based LMM scoring to assess visualization similarity.The code-level measure is called LLM-Score; the chart-level measure is called LMM-Score.Source: §H.1 · p. 56

H.2 Base Evaluation

  • For code-perspective evaluation, visualization elements are extracted from Matplotlib Figure objects and compared between ground-truth and generated code.The paper notes that several plotting libraries rely on Matplotlib’s underlying plotting functions.Source: §H.2 · p. 56

H.2.1 Comparison Between Chartmimic

This section compares the evaluation method with ChartMimic and describes the metrics and assessment procedures used for chart-code evaluation. It also documents evaluated-model configurations, components, licenses, sources, and release dates.

H.2.1 Comparison Between Chartmimic

  • Parsing Figure objects once enables more efficient, version-independent and cross-library evaluation than ChartMimic’s repeated executions, hard-coded rules, and function-pattern matching.The method inspects visual objects such as Line and Patch; ChartMimic may execute both code versions up to twelve times per generated code.Source: §H.2.1 · p. 57
  • The framework adds grid alignment, legend consistency, visual and data parameters, while LLM-Score and Base-Score show high consistency and Base-Score reflects trend-level visual similarity.The passage points to Fig. 6 and Fig. 8 for analysis of code-level and chart-level similarity.Source: §H.2.1 · p. 57

H.2.2 Color Score

  • For color fidelity, the method maps colors to element types and data keys, weights important data elements more heavily, and compares matched colors using normalized reversed Euclidean distance.It maps the resulting weighted similarity to Precision, Recall, and F1-Score.Source: §H.2.2 · p. 60

H.2.3 grid Score

  • Grid similarity compares multisets of per-subplot X- and Y-grid visibility states with F1, ignoring grid-line style to focus on whether grids are present.When the ground-truth chart has no grid descriptors, Recall is defined as 1.0.Source: §H.2.3 · p. 61

H.2.4 Layout score

  • Layout similarity represents each subplot by its structural GridSpec position and size, then compares descriptor multisets to assess correctly generated and placed subplots.Precision penalizes redundant or incorrectly placed subplots, while recall reflects missing required subplots.Source: §H.2.4 · p. 62

H.2.5 Legend score

  • Legend evaluation decomposes legends into entries and matches their text semantics and bounding-box positions under configurable semantic-only or combined constraints.Each entry descriptor records its text and the containing legend object's bounding box.Source: §H.2.5 · pp. 62–63
  • The chart-code evaluator scores data and visual parameters through similarity and matching, while separately assessing chart-type composition and semantically categorized chart text.Text matching uses Levenshtein similarity within semantic categories.Source: §H.2.6 · pp. 63–64, §H.2.8 · p. 65, §H.2.9 · p. 66

H.3 LLM-Evaluation

  • LLM-based code evaluation decomposes code quality into eight independent dimensions to make assessment more focused, diagnostic, and interpretable.The dimensions include data logic, chart type and geometry, color, visual parameters, layout, legend, grid and axes, and text content.Source: §H.3 · pp. 66–67

H.4 LMM-Evaluation

  • The VLM framework compares reference and generated chart images in one holistic evaluation call, starting at 100 and deducting points for visual discrepancies.It is intended to assess differences ranging from overall layout to pixel-level details.Source: §H.4 · p. 67

I Run configurations

  • Proprietary models use official API defaults where possible, with temperature 0.1 and Top-P 0.99999; open-source models use stochastic decoding and context lengths of 8192 tokens for Levels 1–2 and up to 32768 for Level 3.The configurations follow official implementations or documentation where possible to support reproducible evaluation.Source: I Run configurations · p. 67

J Open-Source Model Components

  • The open-source model component summary covers vision encoders, language backbones, and input resolutions.Source: J Open-Source Model Components · p. 67
  • Models use varied ViT or CLIP-based and hybrid visual encoders with different language backbones, while input resolution defaults are retained where possible and constrained for stricter architectures.The run configurations also report unset values as defaults and use model-page code where possible.Source: J Open-Source Model Components · pp. 67, 69, Table 20 · p. 68

K Model License Information

  • Licensing spans proprietary systems and open-source models; “Not Applicable” marks missing public license information, for which users should consult the official provider’s current terms.Model licenses govern pretrained weights or API services, while code licenses cover associated implementations; proprietary terms are typically platform-specific.Source: K Model License Information · p. 69

L Model Source

  • Model sources and release dates are documented for traceability, covering proprietary product pages and open-source repositories across a release timeline from early 2024 to late 2025.Open-source repositories or checkpoints are primarily hosted on platforms such as Hugging Face.Source: L Model Source · p. 69

M Sample of Data

This appendix samples Chart2Code data and implementation materials, including evaluated-model metadata, user instructions, and code for chart reproduction and editing. It also presents chart-generation examples and evaluator fragments for comparing generated charts with references across visual elements and parameters, and for computing data and visual metrics.

M Sample of Data

  • Benchmark model tables summarize open-source models’ visual and language components, evaluation input resolutions, release times, sources, and licenses.For DeepSeekVL-7B and GLM-4-9B, the evaluation applies maximum input-size constraints.Source: Table 21 · p. 70, Table 22 · p. 70, Table 23 · p. 71
  • Reference-image reproduction samples pair Python-code requests with implementations for sunburst and radar charts, including a multi-industry cloud-adoption radar display.The examples include evaluation-capability and evaluation-dataset sunbursts, an evaluation-results radar chart, and per-industry plots.Source: M Sample of Data · pp. 72, 74
  • Level 1 examples request a monthly commodity heatmap from raw data and a styled compensation combination chart based on a table image.The heatmap instruction maps months and three commodity categories; the compensation prompt specifies director names, three compensation measures, and visual styling.Source: M Sample of Data · pp. 75–76
  • Other prompts specify a raincloud-and-histogram layout and a multi-panel analysis of model performance distributions, tool wear, and model comparisons.The raincloud layout uses a top plot spanning two columns and two lower histograms; the analysis prompt includes linked plots and a zoomed comparison.Source: M Sample of Data · pp. 77, 79
  • A dashboard prompt combines CCA lines and CKA bars, a distribution box plot, and a zoomed center-layer view linked by a highlighted region.The requested layout uses a 2×2 GridSpec, with the zoom focused on layers 10–20.Source: M Sample of Data · p. 78
  • Sample instructions span a research-count dashboard, model-performance comparisons, fuel-type and email-length plots, and a mental-health-history box plot.The requests specify chart layouts, visual styling, and data comparisons, illustrating varied chart-generation scenarios.Source: M Sample of Data · pp. 80–84

N.1 color

  • The color evaluator accepts generated and ground-truth figures, extracts their colors, and calculates color metrics; exceptions mark evaluation as failed.Source: §N.1 · p. 86

N.2 Grid

  • The grid evaluator’s callable accepts generated and ground-truth figures and checks whether either figure is missing.Source: §N.2 · p. 87
  • The Grid evaluator records whether x- and y-gridlines are visible and computes precision, recall, and F1 for matched grid settings.If neither figure has grid settings, all three scores are set to 1.0; if only one has grids, they are set to 0.0.Source: §N.2 · p. 87

N.3 Layout

  • The Layout evaluator records each axes’ subplot-grid dimensions and row/column span, then computes precision, recall, and F1 from matched layouts.The extracted layout information is compared between the generated and reference figures.Source: §N.3 · p. 87

N.4 Legend

  • The Legend evaluator extracts visible legend text and bounding boxes, then scores generated legends by matching them to reference legends with overlapping boxes.It reports precision, recall, and F1 for the matches.Source: §N.4 · p. 88

N.5 Visual

  • The Visual evaluator extracts artist parameters from lines and patches, then applies strict parameter metrics while excluding selected styling and label fields.Extracted line attributes include data, linestyle, linewidth, marker, size, and alpha; rectangle patches include position, width, and height.Source: §N.5 · p. 89
  • Visual extraction records polygon-patch vertices and collection properties including alpha, offsets, and sizes.The fragments show these properties being appended to extracted parameters.Source: §N.5 · p. 89
  • Value similarity returns exact matches for close numeric values and uses overlap-based comparisons for array or list values.Numeric values are compared with np.isclose; numeric arrays use intersection-over-union, with a set-overlap fallback for other values.Source: §N.5 · p. 89
  • The strict metric matches generated elements to reference elements by parameter similarity, separates data from visual parameters, and computes precision, recall, and F1 for each category.Element types are excluded from similarity scoring.Source: §N.5 · p. 89

N.6 Data

This section describes the prompts and evaluation procedures for chart generation and assessment, covering chart-data and text extraction, text and chart-type matching, and dimension-by-dimension similarity scoring. It also specifies how generation instructions and reference inputs are used and how evaluation results are structured.

N.6 Data

  • The data evaluator extracts line, patch, and collection attributes, distinguishing rectangle and polygon patches for parameter comparison.Its extracted attributes include line data and styling, patch geometry, and collection offsets or sizes when available.Source: §N.6 · p. 90
  • The data evaluator compares generated and reference element parameters, then reports separate precision, recall, and F1 score blocks for data and visual parameters.Numeric values are compared with np.isclose, while arrays and lists use similarity comparisons including set overlap.Source: §N.6 · p. 91

N.7 Text

  • The text evaluator groups figure text into categories—including titles, axis labels, tick labels, legends, and annotations—and computes strict metrics from categorized text similarity.It also extracts figure-level suptitles and filters out categories with no text.Source: §N.7 · p. 92
  • Text evaluation greedily matches generated strings to unmatched ground-truth strings by Levenshtein ratio, then computes precision, recall, and F1.The matching loop selects the best remaining ground-truth match for each generated string.Source: §N.7 · p. 92

N.8 Type

  • Chart-type evaluation identifies types from Matplotlib artist classes in the generated and reference figures, then calculates strict precision, recall, and F1 over detected types.The detector checks artists such as lines, rectangles, wedges, and collections.Source: §N.8 · p. 93

O.1 generation Prompt

  • One reproduction prompt asks models to infer data from a plot image and generate complete, executable Matplotlib code reproducing its plot type and visual elements.The specified elements include labels, titles, styles, legends, annotations, and gridlines.Source: §O.1 · pp. 93–94
  • Other prompts provide data and instructions alongside a reference image, requiring models to use the supplied data and reproduce the reference style in self-contained code.A separate template distinguishes a data-source image from a style-reference image.Source: §O.1 · p. 94
  • Additional prompts ask models to extract data from a source image while applying a separate style reference, or to edit a base plot according to instructions and return executable code.The editing prompt requires the generated script to produce the modified visualization without external data files.Source: §O.1 · pp. 95–96
  • The generation prompt requests complete, executable Python using embedded CSV data extracted from Excel, task instructions, and a reference image’s visual style.It directs models to define the data in the code rather than depend on external files.Source: §O.1 · p. 96

O.2 LLM-Score Prompt

  • The code-based score judges deterministically rendered similarity through static analysis, deducting for uncertain outputs and mismatches in data, chart geometry, colors, and visual parameters.The rubric also specifies per-dimension scores starting at 100 and requires deductions for visible deviations.Source: §O.2 · pp. 97–98
  • The code-based rubric separately assesses layout, legends, grids and axes, and text content in its rendered-output dimensions.Its output format requests a score and brief concrete reason for each dimension.Source: §O.2 · p. 98
  • The image-based prompt compares ground-truth and generated charts using visual information only, evaluating data fidelity, layout, color and style, and chart type.It treats minor numeric drift without trend change as a minor error, while wrong trends, scales, or correspondence are fatal.Source: §O.2 · p. 99
  • Wrong chart type is fatal, and the prompt specifies an automatic zero for three or more dimensions with major errors.It also states that any error in any dimension triggers a hard score cap.Source: §O.2 · p. 100
  • The prompt maps scores from 90–100 to visually identical charts, 70–89 to slight deviations, and 0 to fundamentally incorrect results.It also assigns distinct interpretations to intermediate ranges, including severe mismatch for 1–9.Source: §O.2 · p. 100
  • Evaluators start at zero, identify visible differences, classify them by severity, assign penalties, and do not compensate across dimensions.When uncertain, the prompt directs evaluators to lower the score and forbids inflating scores for visual similarity despite data errors.Source: §O.2 · p. 100
  • The required output is a valid JSON object containing a final score, summary, errors by dimension, and scores with reasons for layout, color/style, and chart type.The specified format prohibits Markdown code blocks.Source: §O.2 · p. 100

Related papers

Loading 2510.17932v4…