PaperlayerView exact revision on arXiv

Source-linked AI summary

RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang, Qingbin Li, Shannan Yan, Zhetong Li, Changguo Jia, Junfei Wu, Zilei Wang, Qiang Liu, Liang Wang

arXiv:2603.25804v1cs.CL

TL;DR

VLMs have been insufficiently evaluated on complex chart-to-code generation from real-world data and on iterative refinement. RealChart2Code addresses this gap with a 2,896-instance benchmark spanning replication, raw-data reproduction, and conversational refinement, then evaluates 14 leading models. The results show substantial degradation on complex multi-panel visualizations, with proprietary models generally outperforming open-weight models.

  • Problem

    Existing chart-to-code benchmarks provide limited evidence about VLM performance on complex multi-panel visualizations, authentic large-scale data, and multi-turn refinement.

  • Method

    RealChart2Code benchmarks chart replication, reproduction from raw data, and iterative refinement using 2,896 authentic-data instances, and evaluates 14 VLMs.

  • Results

    The evaluation finds significant performance degradation on complex real-world tasks and a distinct capability gap between proprietary and open-weight models.

  • Takeaways & Limitations

    RealChart2Code provides a more discriminating evaluation of complex visual reasoning and code generation than simpler benchmarks.

  • Takeaways & Limitations

    The implementations are confined to Matplotlib, and model-based judges may miss subtle visual artifacts such as minor overlaps or precise color nuances.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains. However, their ability to replicate complex, multi-panel visualizations from real-world data remains largely unassessed. To address this gap, we introduce \textbf{\texttt{RealChart2Code}}, a new large-scale benchmark with over 2,800 instances grounded in authentic datasets and featuring tasks with clear analytical intent. Crucially, it is the first benchmark to systematically evaluate chart generation from large-scale raw data and assess iterative code refinement in a multi-turn conversational setting. Our comprehensive evaluation of 14 leading VLMs on \texttt{RealChart2Code} reveals significant performance degradation compared to simpler benchmarks, highlighting their struggles with complex plot structures and authentic data. Our analysis uncovers a substantial performance gap between proprietary and open-weight models and confirms that even state-of-the-art VLMs often fail to accurately replicate intricate, multi-panel charts. These findings provide valuable insights into the current limitations of VLMs and guide future research directions. We release the benchmark and code at \url{https://github.com/Speakn0w/RealChart2Code}.

1 Introduction

Chart-to-code models remain weak on complex visualizations grounded in authentic data, while existing benchmarks insufficiently test these capabilities. RealChart2Code addresses this gap with a large-scale benchmark and evaluation of 14 VLMs.

  • Current VLMs can generate simple single-panel charts but struggle with multi-panel layouts and large, complex structured data.
  • Prior benchmarks rely mainly on simple charts, internet chart-code pairs, or synthetic data and omit multi-turn code-refinement evaluation.
  • RealChart2Code contains 2896 instances using authentic datasets, intricate chart structures, interactive workflows, and three tasks covering visualization generation, semantics, and modification.
  • The benchmark was evaluated on 14 prominent VLMs, with human inspection and correlation analysis used to validate the quantitative evaluation.
  • Performance analysis across benchmarks and qualitative case studies reveals differing capabilities across tasks of varying difficulty and types.

2 Related Works

Chart-to-code research sits within broader chart understanding and text-to-visualization work, but existing benchmarks predominantly use simple, single-panel plots. RealChart2Code targets the resulting gap with complex multi-panel charts derived from real-world data.

  • Prior LLM visualization research includes chart understanding, Text-to-Visualization, and Chart-to-Code tasks.
  • The benchmark’s core tasks are Chart Replication, Chart Reproduction with raw data, and conversational Chart Refinement.
  • Existing chart-to-code benchmarks predominantly feature simple, single-panel plots that inadequately test complex layouts and high information density.
  • RealChart2Code is designed to evaluate intricate, multi-panel charts derived from real-world data.

3 RealChart2Code

RealChart2Code defines chart-to-code as executable conditional code generation and evaluates three increasingly demanding tasks. Its benchmark combines diverse complex visualizations, authentic data, manually implemented references, injected errors, and multi-level evaluation.

  • 3.1 Task Definition: Given a chart image V and prompt P, the model generates executable code C according to C = F(V, P).
  • 3.1 Task Definition: Chart Replication uses the image alone, Chart Reproduction adds raw data and metadata, and Chart Refinement requires multi-turn debugging from user instructions.
  • Chart Types: The benchmark covers seven visualization-intent categories, 50 plot types, and complex composite or multi-panel layouts.
  • Dataset Distribution: RealChart2Code spans eight domains and 35 fine-grained sub-topics, with dispersed image and CSV distributions indicating substantial diversity.
  • Construction Pipeline: Its four-stage construction pipeline consists of data collection and filtering, visualization task design, code implementation, and error injection.
  • Data Collection and Filtering: The curated data process selected 1,036 high-quality datasets from an initial screening of more than 8,000 datasets.
  • Visualization Task Design: The dataset includes 1,016 unique visualizations producing 2,032 replication and reproduction instances, plus 864 refinement tasks, for 2,896 total instances.
  • Evaluation: Evaluation first measures executable-code Pass Rate, then scores valid outputs on eight visual criteria, with reproduction data consistency checked programmatically.

4 Experiments

The experiments evaluate 14 models across RealChart2Code and two established benchmarks, revealing substantially weaker performance on complex, authentic-data tasks and a large proprietary–open-weight gap.

  • Experimental Setup: 14 models are evaluated on RealChart2Code, ChartMimic, and Plot2Code using standardized API-based experiments averaged across three runs.The RealChart2Code evaluation covers Chart Replication, Chart Reproduction, and Chart Refinement.
  • Cross-Benchmark Analysis: Models scoring above 75 on ChartMimic can fall to 3.6 and 3.4 on RealChart2Code, showing that simpler benchmarks do not guarantee complex-task success.RealChart2Code is reported to differentiate model capabilities more strongly than prior benchmarks.
  • Evaluation Reliability: The evaluation framework is supplemented by reliability analysis using Fleiss’ κ for inter-agent agreement and Cohen’s κ for judge–human agreement.Table 4 reports these two agreement analyses across eight metrics.

5 Discussion

The discussion validates the automated evaluation and analyzes where models fail, showing distinct error patterns and severe degradation when moving from simplified benchmarks to realistic visualization tasks.

  • Reliability Analysis: The multi-agent evaluation achieves an average Inter-Agent κ of 0.8239, indicating high stability across the benchmark.This agreement is measured with Fleiss’ κ across the entire RealChart2Code benchmark.
  • Reliability Analysis: Human experts and the automated judge reach an average Cohen’s κ of 0.83 on 600 sampled tasks from Claude-4.5-Sonnet results.Figure 5 presents the corresponding score distributions and 95% confidence intervals.
  • Error Analysis: Open-weight models frequently produce syntax and execution errors, then exhibit layout and structural failures when their code runs.Examples include hallucinated libraries, invalid functions, overlapping subplots, and incorrect grid definitions.
  • Error Analysis: Proprietary models generally avoid syntax errors but predominantly fail through data mapping errors involving incorrect axes or visual attributes.These failures can occur despite otherwise correct visual structure.
  • Iterative Refinement: Regressive Editing occurs when a requested refinement fixes one part of the code but introduces new errors in previously correct components.This reflects difficulty balancing local updates with global consistency during multi-turn conversations.
  • Performance Across Benchmarks: Gemini-3-Pro-Preview declines from scores exceeding 91% on existing benchmarks to approximately 50% on RealChart2Code, while Qwen3-VL drops from approximately 85% to below 25%.The comparison suggests limited transfer from simplified synthetic environments to complex, data-driven visualization tasks.

6 Conclusion and Future Works

The paper introduces RealChart2Code to evaluate VLMs on realistic, complex visualization tasks and finds substantial degradation on multi-panel layouts and real-world data.

  • Conclusion: RealChart2Code is a large-scale benchmark grounded in authentic datasets and covering chart replication, raw-data reproduction, and iterative refinement.The benchmark evaluates 14 leading models across these capabilities.
  • Conclusion: Current models perform well on simple plotting but degrade substantially on complex multi-panel layouts and real-world data.Proprietary models show superior visual reasoning, whereas open-weight models frequently struggle with syntax and spatial logic.
  • Future Work: Future work will investigate automated generation of high-quality synthetic data to address scarce complex training examples and improve generalization on intricate layouts.

7 Limitations

RealChart2Code is limited to Matplotlib, and its automated visual judges may miss subtle artifacts despite correlating strongly with human experts.

  • Implementations are confined to Matplotlib, which the authors argue provides a robust proxy for general plotting skills through granular, imperative code.The scope is justified as reflecting fundamental visualization understanding and logical reasoning.
  • MLLM-based judges may miss subtle visual artifacts, including minor element overlaps and precise color nuances.The authors identify fine-grained visual evaluation as a direction for future work.

A Benchmark Details

RealChart2Code is designed to test visual understanding and code generation on authentic, complex visualizations rather than simplified charting tasks. It combines diverse chart structures with image-grounded replication and iterative refinement.

  • The benchmark contains 50 chart types and counts each unique visualization task once across single-turn and multi-turn interactions.This distribution is summarized in Table 5.
  • A Docker-based sandbox with Python 3.13 and data-analysis and visualization libraries standardizes and isolates execution of model-generated code.The environment includes Pandas, NumPy, and Matplotlib.
  • RealChart2Code uses real-world datasets to expose models to the noise, scale, and irregularity of actual data-science workflows.The visualizations are intended to reflect genuine analytical intent rather than arbitrary plotting.
  • The benchmark evaluates visual-grounded reverse engineering by requiring models to translate a reference image’s layout, styling, and color mapping into executable Matplotlib code.This differs from open-ended text-to-visualization generation because the model must reproduce a visual reference faithfully.
  • Over 50 chart types and composite layouts such as multi-panel figures and dual-axis plots target the complexity gap in current evaluations.These structures probe spatial reasoning and understanding of plotting-library constraints.
  • A Chart Refinement module uses injected errors to assess whether models can diagnose issues, interpret feedback, and modify existing code without breaking functionality.The module is intended to reflect the iterative nature of real-world coding and collaborative assistance.

B.2 Data Collection and Filtering

The benchmark is built through large-scale dataset screening, expert curation, visualization design, manual code implementation, and controlled error injection. Quality checks enforce executable, reproducible, and logically solvable tasks.

  • Data Collection and Filtering: The curation pipeline begins with Kaggle repositories and uses scientific licensing terms to filter open-source data into a high-quality benchmark collection.The process is organized into automated screening followed by expert curation.
  • Data Collection and Filtering: Over 8,000 candidate datasets containing more than 100,000 files and roughly 30 billion rows were screened using votes, downloads, and usability ratings.This filtering removed incomplete, poorly documented, or trivial datasets.
  • Data Collection and Filtering: Expert review selected 1,036 datasets, yielding 3,271 raw files and approximately 860 million rows for data-intensive generation tasks.Selection emphasized data richness and domain diversity.
  • Visualization Task Design: The curation produced 1,016 visualization scenarios and 2,032 replication and reproduction instances across more than 50 chart types.The scenarios are based on the curated datasets and taxonomy of analytical intents.
  • Visualization Task Design: The benchmark taxonomy spans seven visualization intents, including correlation, deviation, ranking, distribution, composition, change, and groups.These intents cover relationships, variation, ranking, spread, part-to-whole structure, temporal trends, and clusters.
  • Code Implementation: Five expert Python developers manually implemented all 1,016 visualizations, and code was tested for executable, reproducible, and idiomatic behavior.The implementations use Matplotlib with standard data-processing libraries rather than model-generated ground truth.
  • Error Injection: The refinement subset contains 864 tasks created by injecting visual, data-mapping, and chart-type errors into selected implementations.Models receive flawed code, its incorrect chart, and a natural-language correction instruction in a multi-turn setting.
  • Quality Control: Refinement tasks were accepted only after checks of error visibility and solvability, followed by consensus procedures that discarded unresolved cases.The protocol included peer review, execution checks, and unanimous pass requirements.

C Evaluation Details

Evaluation measures visual fidelity for replication and refinement, while reproduction additionally verifies computational data alignment. Metrics use three-level scores to assess chart structure and correctness.

  • The scoring rubric has two dimensions: Visual Structure Alignment for reference-chart fidelity and Data Alignment for computational correctness in code reproduction.The framework covers all three benchmark tasks.
  • Chart Replication and Chart Refinement use 0/1/2 scoring for visual structure alignment against the reference chart.The rubric evaluates structural elements rather than only overall appearance.
  • Visual Structure Alignment: Chart-type scoring distinguishes exact matches across subplots, matches with minor secondary-element differences, and fundamental type mismatches.Examples include dual-axis and combination charts.
  • Visual Structure Alignment: Layout scoring evaluates subplot-grid dimensions, positioning, and element placement, with full credit requiring placement within ±2% of reference dimensions.Lower scores allow minor deviations or penalize incorrect grids and major misplacements.
  • Visual Structure Alignment: Additional visual metrics assess textual content, axis and legend properties, color mappings, stylistic attributes, and completeness of visual components.These criteria cover wording, scales, colors, markers, fonts, legends, annotations, and overlays.
  • Data Alignment: Data-pattern scoring compares positional accuracy, trends, distributions, and statistical markers, while Task 2 replaces it with code-level Data Alignment.Data Alignment verifies equivalent files, column selections, transformations, and mappings rather than visual similarity.

D Evaluation Prompts

The evaluation prompts require models to analyze authentic data and reference images, generate executable Matplotlib or Seaborn code, and reproduce visual, structural, and data properties. A structured JSON judging protocol scores fidelity across chart structure, layout, text, axes, colors, style, data patterns, completeness, and execution quality.

  • D Evaluation Prompts: The appendix also specifies self-contained executable output, equivalent-data checks, and prompt formats for chart replication and reproduction.Equivalent operations are accepted, but both codes must load the same files and access the same fields.
  • Evaluation Prompt for Chart Replication: Chart evaluation scores visual-structure alignment through chart type, spatial layout, text, axis, color, style, data-pattern, and component-completeness criteria.Each category uses a three-level scoring scheme, with criteria ranging from exact matches to missing or incorrect elements.
  • Evaluation Prompt for Chart Replication: The evaluation separately checks text, axes, colors, style, data patterns, and component completeness against the reference chart.Examples include exact wording, axis properties, identical color mappings, stylistic attributes, recognizable data trends, and presence of critical components.
  • Evaluation Prompt for Chart Replication: Technical execution is evaluated independently for overlap, composition, text legibility, and spatial balance.The scoring criteria distinguish zero overlap and optimal distribution from critical overlap, poor composition, or severely undersized text.
  • Evaluation Prompt for Chart Replication: The evaluator must return a single valid JSON object containing scores, reasons, and improvement recommendations for each evaluation category.The required output includes visual-structure, execution-quality, compositional-balance, and recommendation fields.
  • Generation Prompt for Chart Reproduction: Models must analyze datasets, reference images, columns, transformations, labels, relationships, and displayed trends before generating replication code.Inputs may include dataset schemas, sample rows, multiple CSV/XLSX files, and a target visualization image.
  • Generation Prompt for Chart Reproduction: Generated code must load the provided files, use actual columns and data types, apply required preprocessing, and avoid synthetic data.The prompt mandates pandas-based file loading and prohibits generating synthetic data.
  • Generation Prompt for Chart Reproduction: Replication requirements cover chart types, subplot arrangement, figure dimensions, axes, data patterns, legends, statistical elements, colors, typography, styles, grids, and annotations.The prompt specifies pixel-level fidelity using Matplotlib and/or Seaborn exclusively.

E.1 Performance Analysis Across Benchmarks

RealChart2Code exposes a substantial complexity gap: models that perform strongly on simple benchmarks lose much of their performance on authentic, complex, multi-panel tasks. The degradation is especially pronounced for open-weight models.

  • The Complexity Gap: 50.6% and 51.3% are RealChart2Code scores for Gemini-3-Pro-Preview and Claude-4.5-Opus, down from 96.0% and 91.2% on existing tasks.The paper describes this near-halving as the Complexity Gap.
  • The Complexity Gap: Existing benchmarks can saturate on top-tier proprietary models while failing to distinguish robust reasoning for complex visualizations from competence with simple syntax.RealChart2Code is presented as a more demanding test of authentic-data visualization and iterative refinement.

F Case Study

The case studies show that models often reproduce local plotting details but fail to assemble dense visualizations globally. Errors include poor layout planning, hierarchical decomposition, canvas scaling, semantic grouping, and invalid API usage.

  • Performance Collapse on Real-World Tasks: Across models, performance on RealChart2Code falls far below parity with existing benchmarks, with drops approaching 50%.Figure 7 plots normalized existing-benchmark scores against RealChart2Code scores and marks equal performance with y=x.
  • Error Case 1: Models can reproduce individual subplot styles and data distributions but fail to synthesize them into a coherent multi-panel layout.Default subplot positioning causes overcrowding and overlapping labels, whereas explicit layout management is needed.
  • Error Case 2: Nested dashboards expose failures in hierarchical spatial reasoning, producing fragmented subplots, missing elements, and collapsed global structure.The examples involve scatter matrices, joint plots, marginal distributions, dual-axis overlays, and other heterogeneous charts.
  • Error Case 3: Improper figure sizing can shrink an otherwise constructed subplot grid to thumbnail scale inside an oversized canvas, leaving excessive whitespace.The mismatch involves figure dimensions, element scaling, resolution, and aspect ratio.
  • Error Case 4: Models may flatten Joint-Grid-style composites by treating marginal axes as independent panels rather than spatially linked components.This breaks the semantic grouping and cohesive narrative of the reference visualization.
  • Error Case 5: Approximately 20% of Qwen3-VL-235B solutions contain plausible but invalid API calls such as the nonexistent seaborn-v0_11 style.These failures are attributed to library-knowledge errors rather than visual-reasoning deficits.
Loading 2603.25804v1…