PaperlayerView exact revision on arXiv

Source-linked AI summary

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code

Timur Galimzyanov, Sergey Titov, Yaroslav Golubev, Egor Bogomolov

arXiv:2412.02764v2cs.SEcs.AIcs.LG

TL;DR

LLM plotting benchmarks lack broad evidence for DataFrame-based visualization, instruction compression, and library differences. This paper introduces the 175-task PandasPlotBench and evaluates models across plotting libraries and task formulations. Shortening tasks has minimal impact with structured DataFrame descriptions, whereas Plotly remains difficult, with approximately 22% failed code cases.

  • Problem

    Existing benchmarks provide limited evidence for accurate plotting-code generation from brief instructions and DataFrame descriptions.

    Source: §I · p. 1

  • Method

    PandasPlotBench is a human-curated, synthetically generated benchmark with 175 Pandas DataFrame visualization tasks and multiple task lengths.

    Source: §I · p. 1, B. Data Characteristics · p. 2, §VI · p. 4

  • Results

    Task compression minimally affects plotting capability, while Plotly remains difficult relative to Matplotlib and Seaborn, with approximately 22% failed code cases.

    Source: §I · p. 1, §VI · p. 4

  • Takeaways & Limitations

    The benchmark supports research on improving user experience in data visualization and analysis through comparisons across models, task formulations, and libraries.

    Source: §VI · p. 4

    The dataset is limited to 175 Matplotlib-gallery-derived data points and mostly concise DataFrames, which may not represent real-world data complexity.

    Source: §V · p. 4

Abstract

from arXiv · show

This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope. The dataset includes 175 unique tasks. Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly. We show that the shortening of tasks has a minimal effect on plotting capabilities, allowing for the user interface that accommodates concise user input without sacrificing functionality or accuracy. Another of our findings reveals that while LLMs perform well with popular libraries like Matplotlib and Seaborn, challenges persist with Plotly, highlighting areas for improvement. We hope that the modular design of our benchmark will broaden the current studies on generating visualizations. Our dataset and benchmark code are available online: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench; https://github.com/JetBrains-Research/PandasPlotBench.

I. INTRODUCTION

PandasPlotBench addresses limited evaluation of LLM-generated plotting code by providing a broad, human-curated benchmark for visual data exploration. Experiments show that task compression has minimal effect with structured DataFrame descriptions, while Plotly remains more challenging than Matplotlib and Seaborn.

Motivation

  • Existing benchmarks provide limited coverage of DataFrame-based visualization, instruction rephrasing, plotting-library differences, and broad model comparisons.MatPlotBench is identified as the only prior work addressing visualization from a DataFrame or CSV, with only 25 CSV data points.Source: §I · p. 1

Contribution

  • PandasPlotBench provides 175 synthetic, human-curated tasks for generating plotting code from Pandas DataFrames and natural-language instructions.The benchmark is designed to reduce direct links between task answers and training data.Source: §I · p. 1

Results

  • Task compression minimally affects plotting capability when models receive well-structured DataFrame descriptions.The study examines concise instructions as a proxy for real-world user behavior.Source: §I · p. 1
  • Models perform differently across libraries, with Plotly code generation remaining difficult relative to Matplotlib and Seaborn.The benchmark evaluates Matplotlib, Seaborn, and Plotly.Source: §I · p. 1

Resources

  • The dataset, baseline implementations, evaluation code, prompts, scores, and generated plots are made available online.The benchmark is hosted online, with implementation and evaluation code on GitHub.Source: §I · p. 1

II. PANDASPLOTBENCH

PandasPlotBench is constructed by converting validated Matplotlib gallery scripts into paired DataFrames and plotting tasks. Multiple automated and manual validation stages reduce the original gallery collection to 175 final data points.

A. Data Collection and Processing

  • The construction pipeline reduces 501 Matplotlib gallery scripts to 175 validated DataFrame-and-task data points.The process filters executable scripts, splits data-generation and plotting code, manually verifies the splits, and validates final plots.Source: A. Data Collection and Processing · pp. 1–2
  • GPT-4 [22] separates each script into a DataFrame-generating component and a plotting component before manual verification.The resulting DataFrames are saved as CSV files after review.Source: A. Data Collection and Processing · p. 2
  • GPT-4V generates detailed, compressed, and single-sentence plotting instructions from plotting code, DataFrame metadata, and plot images.All generated tasks are manually verified.Source: A. Data Collection and Processing · p. 2

B. Data Characteristics

Each benchmark item represents a user asking a model to plot a Pandas DataFrame and includes data, preparation code, a reference plot, reference Matplotlib code, and multiple instruction lengths.

B. Data Characteristics

  • Each of the 175 data points simulates a user with a Pandas DataFrame who wants a plot from a natural-language task description.The dataset is designed around typical data-visualization requests.Source: B. Data Characteristics · p. 2
  • Every item includes a CSV file, loading and preparation script, ground-truth plot, and ground-truth Matplotlib visualization code.Preparation may include minimal transformations such as converting string fields to objects.Source: B. Data Characteristics · p. 2
  • Each plotting task has detailed, 2–3-sentence compressed, and single-sentence versions.Detailed tasks include plot and style descriptions, whereas compressed tasks omit plot style.Source: B. Data Characteristics · p. 2

C. Evaluation Metrics

The benchmark executes generated code in isolated Jupyter cells and evaluates outputs with both visual similarity and task adherence. Because text tasks can omit visual details, the authors prefer task-based scoring while retaining both measures.

C. Evaluation Metrics

  • Generated code is executed in individual Jupyter cells, and incorrect-code rate is the proportion of cells that produce no plot.Errors are ignored when a cell returns both a plot and errors.Source: C. Evaluation Metrics · p. 2
  • Visual scoring rates generated-versus-ground-truth plot similarity on a 0–100 scale, while task-based scoring rates adherence to the task description.Both scores are assigned by the multimodal GPT-4o Judge model.Source: C. Evaluation Metrics · p. 2
  • Task-based scoring is preferred because generated plots need not exactly match underspecified ground-truth images and may sometimes improve on them.The two scoring approaches have Pearson correlation 0.58 for the best-performing model.Source: C. Evaluation Metrics · p. 2
  • CodeBERT Score was discarded because it showed no correlation with visual or task scores and could not assess implementation-equivalent solutions reliably.Correct and incorrect code could target the same task while differing only in implementation details.Source: C. Evaluation Metrics · p. 2
  • Human scores correlate more strongly with task-based scores than visual scores, with Pearson coefficients of 0.85 and 0.66, respectively.The authors describe this as preliminary evidence and call for evaluation with multiple experts.Source: C. Evaluation Metrics · p. 2

III. EXPERIMENTS

The experiments compare prompting setups, plotting libraries, and task lengths, using mean scores and good-score ratios to evaluate generated visualizations. They also test concise code-only prompting and repeat each factor five times for robustness.

III. EXPERIMENTS

  • The benchmark evaluates DataFrame prompting, proprietary and open models, three plotting libraries, and task-length variants across repeated runs.The best-performing model was selected for subsequent experiments, and each factor was run five times.Source: §III · p. 3
  • Prompting models to return fewer explanations and only commented code did not significantly affect benchmark scores.The change was intended to increase code-generation speed.Source: §III · p. 3
  • Mean visual and task-based scores, plus the ratios of scores at least 75, summarize each model’s benchmark performance.Source: §III · p. 3

A. Benchmarking Models

The model comparison identifies GPT-4o and Claude 3.5 Sonnet as the highest-scoring proprietary models, while large Llama models are comparable and smaller ones often fail to produce functional code. GPT-4o was chosen for later experiments because it combined high performance with greater speed.

A. Benchmarking Models

  • Including the first five DataFrame rows with column names and types produced the best prompting performance among the tested descriptions.This setup uses the head(5) method supplemented with schema information.Source: A. Benchmarking Models · p. 3
  • GPT-4o and Claude 3.5 Sonnet share the highest scores, while GPT-4o is faster and was selected for subsequent experiments.Claude 3.5 Sonnet is 20% slower because of longer responses; Claude 3 Haiku is fastest but scores significantly lower.Source: A. Benchmarking Models · p. 3
  • Large Llama models, including 70B and 405B variants, perform comparably to proprietary LLMs, whereas smaller Llama models often fail to produce functional code.Source: A. Benchmarking Models · p. 3

B. Benchmarking Plotting Libraries

The library comparison finds strong performance for Matplotlib and Seaborn but substantially greater difficulty with Plotly. Plotly errors primarily reflect incorrect use of its library API.

B. Benchmarking Plotting Libraries

C. Alternating Task Length

Shortening task descriptions only slightly reduces plotting scores when DataFrame descriptions and other instructions remain intact. Removing the task and using a generic plotting instruction instead makes only one-third of plots satisfactory.

C. Alternating Task Length

  • Significant task shortening only slightly decreased scores, whereas replacing the task with “Plot a given dataframe” left only one-third of plots satisfactory.Shortened tasks were 2–3 times shorter, while the rest of the prompt averaged around 1600 symbols.Source: C. Alternating Task Length · p. 3
  • A single-sentence user task can produce positive results when automatically generated DataFrame descriptions and thorough instructions are provided.Source: C. Alternating Task Length · p. 3
  • Table III reports benchmark scores for different task formulations.Source: Table III · p. 4

IV. RELATED WORK

Existing visualization benchmarks cover code generation, image-guided plotting, and data-analysis queries, but differ in data availability, task complexity, and extensibility. PandasPlotBench addresses these gaps with a modular framework for evaluating plotting across styles, frameworks, and languages.

Benchmark landscape

  • Existing benchmarks span concise Matplotlib snippets, visually guided generation, and SQL- or visualization-language-based data analysis, but provide limited coverage of DataFrame-driven plotting.MatPlotBench includes only 25 data files, while Plot2Code lacks data files and focuses primarily on image-guided plotting.Source: §IV · p. 4

Related datasets

  • ChartMimic provides 1,000 image-to-code examples, whereas nvBench offers 153 data files and 7,274 visualizations for more complex SQL- and Vega-Lite-like tasks.Source: §IV · p. 4
  • DS-1000 contains 1,000 data-science problems, but its solutions average three lines and are too simplistic for assessing practical plotting by modern LLMs and agents.Source: §IV · p. 4

Extensible framework

  • The proposed framework separates setup and framework, task, and styling instructions, enabling evaluation across visualization styles, frameworks, and programming languages.The paper demonstrates this extensibility through experiments on library choice and task verbosity.Source: §IV · p. 4

V. LIMITATIONS AND FUTURE WORK

The benchmark reduces direct training-data memorization risk through synthetic task and data generation, but remains constrained by possible model and source biases. Future work targets broader data sources, more realistic DataFrames, additional languages, and stronger expert validation.

Data and model bias

  • Synthetic tasks and data reduce direct links to training repositories, although modified public Matplotlib ground-truth code and OpenAI-generated tasks may still introduce bias.The authors consider the OpenAI bias negligible and the memorization risk lower than for directly reused benchmark answers.Source: §V · p. 4

Dataset scope

  • The dataset contains 175 data points from the Matplotlib gallery, limiting source diversity and requiring Matplotlib ground-truth plots to evaluate other libraries.Future work proposes expanding to other sources.Source: §V · p. 4
  • Most DataFrames contain only concise plotting data rather than realistic additional columns, so enrichment could make tasks more representative and complex.Source: §V · p. 4

Evaluation scope

  • The benchmark is restricted to Python, and manual scoring was performed only by the first author, motivating extensions to other languages and broader expert validation.Source: §V · p. 4

VI. CONCLUSION

PandasPlotBench is a human-curated benchmark for evaluating LLMs as visual-data-exploration assistants on Pandas DataFrames. Across varied models, task descriptions, and libraries, shortening instructions had little effect when DataFrame descriptions were automatically generated, while Plotly remained substantially more difficult than Matplotlib and Seaborn.

Conclusion

  • PandasPlotBench evaluates AI models as visual-data-exploration assistants by generating plots from Pandas DataFrames using synthetically generated tasks designed to reduce data leakage.Source: §VI · p. 4
  • LLMs produce compilable, high-scoring code for Matplotlib and Seaborn, but approximately 22% of Plotly cases fail.Source: §VI · p. 4
  • Significant task shortening does not substantially affect plotting capability when models receive automatically generated DataFrame descriptions, supporting concise user interfaces.Source: §VI · p. 4
  • State-of-the-art proprietary LLMs and Llama 3.1 70B and 405B achieve high scores, but less popular visualization libraries remain an area for improvement.Source: §VI · p. 4
Loading 2412.02764v2…