Source-linked AI summary
MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization
Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, Maosong Sun
TL;DR
Scientific data visualization with LLMs remains relatively unexplored, despite its research role and the time required to produce informative figures. This paper introduces MatPlotAgent and MatPlotBench, and reports improved LLM plotting performance alongside automatic evaluation scores that strongly correlate with human assessments.
Problem
LLM-based scientific data visualization remains relatively unexplored, while producing informative visualizations is time-consuming and labor-intensive.
Source: §1 · p. 1Method
MatPlotAgent combines code LLMs and multimodal LLMs for query understanding, iterative code debugging, and visual feedback, alongside a 100-case human-verified benchmark.
Source: §1 · p. 2, §4.2 · p. 5Results
Takeaways & Limitations
MatPlotBench supports automated quantitative evaluation, and its scores strongly align with human assessment.
Source: §1 · p. 2, §7 · p. 8Because MatPlotBench targets general scientific visualization, it may not encompass domain-specific requirements and may have limited applicability in some fields.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Scientific data visualization plays a crucial role in research by enabling the direct display of complex information and assisting researchers in identifying implicit patterns. Despite its importance, the use of Large Language Models (LLMs) for scientific data visualization remains rather unexplored. In this study, we introduce MatPlotAgent, an efficient model-agnostic LLM agent framework designed to automate scientific data visualization tasks. Leveraging the capabilities of both code LLMs and multi-modal LLMs, MatPlotAgent consists of three core modules: query understanding, code generation with iterative debugging, and a visual feedback mechanism for error correction. To address the lack of benchmarks in this field, we present MatPlotBench, a high-quality benchmark consisting of 100 human-verified test cases. Additionally, we introduce a scoring approach that utilizes GPT-4V for automatic evaluation. Experimental results demonstrate that MatPlotAgent can improve the performance of various LLMs, including both commercial and open-source models. Furthermore, the proposed evaluation method shows a strong correlation with human-annotated scores.
1 Introduction
Scientific plotting remains time-consuming and labor-intensive, while LLM use for scientific visualization remains relatively unexplored. MatPlotAgent and MatPlotBench address this area with a plotting agent and a human-verified evaluation benchmark.
1 Introduction
- Scientific visualization conveys complex information and can help researchers uncover implicit patterns, but producing informative figures remains labor-intensive.The introduction contrasts this task with text-to-image systems that predominantly focus on artistic expression rather than scientific clarity and precision.Source: §1 · p. 1
- MatPlotAgent combines query understanding, code generation with iterative debugging, and visual feedback that guides code correction.The framework can be driven by code LLMs and multimodal LLMs.Source: §1 · p. 2
- MatPlotBench provides 100 human-verified test cases for evaluating scientific data visualization approaches.Each case includes a user query, input data, and a ground-truth figure.Source: §1 · p. 2
- The proposed GPT-4V scoring mechanism strongly correlates with human-annotated scores, supporting its reliability for evaluating generated figures.The authors use generated figures, ground-truth figures, and human evaluations to assess the correlation.Source: §1 · p. 2
- MatPlotAgent improves the plotting performance of a wide range of LLMs through its proposed visual feedback mechanism.The framework is model-agnostic and uses code LLMs and multimodal LLMs.Source: §1 · p. 2
2 Task Description
The task is to generate a figure that satisfies a textual user query, given the corresponding data. The query can specify plot type, plotted data, structural or spatial requirements, and aesthetic preferences.
2 Task Description
- The system maps a text query and its corresponding data to a figure that satisfies the user’s request.The AI system may be an LLM or an LLM-based agent.Source: §2 · p. 3
- A query can specify visualization type, which data to plot, structural or spatial requirements, and aesthetic preferences.Requirements may concern individual plot elements or the plot as a whole.Source: §2 · p. 3
- The input data is a collection of data points, supplied by the user or stored in an external data file.Source: §2 · p. 3
3 MatPlotBench
MatPlotBench is designed for complex, realistic scientific visualization tasks and uses GPT-4V to score generated figures against ground truths. The authors report strong correlation between automatic and human scores.
3 MatPlotBench
- MatPlotBench targets complex visualization problems closer to real-world scenarios than the short Matplotlib coding problems in DS-1000.The authors note that DS-1000 solutions average only three lines.Source: §3 · p. 3
3.1 Data Collection
- The collection principles call for diverse plot types, representative scientific data complexity, and a balance of easy and challenging cases.Source: §3.1 · p. 3
- The 100 source examples come from the Matplotlib Gallery and OriginLab GraphGallery, with 75 and 25 examples respectively.The sources span common and rare plot types, including advanced examples such as Sankey diagrams and chord diagrams.Source: §3.1 · p. 3
- Queries are drafted from gallery code or images with GPT-4 or GPT-4V, and Matplotlib data are replaced to address observed GPT-4 memorization.OriginLab data are not replaced because they are complex and GPT-4 does not exhibit memorization with those examples.Source: §3.1 · p. 3
- Two experienced annotators refine each query, and ground-truth figures are manually replotted for examples whose Matplotlib data were changed.Unchanged OriginLab images are extracted from the website as ground truths.Source: §3.1 · pp. 3–4
- Three researchers verify that each query aligns with its ground truth, producing 100 high-quality query, data, and figure triples.They check figure elements against query descriptions and correct omissions, ambiguities, redundancies, and errors.Source: §3.1 · p. 4
3.2 Automatic Quantitative Evaluation
- GPT-4V scores generated visualizations from 0 to 100 using the corresponding ground-truth figures as references.This automatic evaluation is proposed to reduce manual evaluation effort and broaden the benchmark’s research use.Source: §3.2 · p. 4
- r=0.876 for GPT-4 and r=0.836 for GPT-3.5; GPT-4V automatic scores were strongly correlated with human-evaluated scores.The correlation analysis repeatedly sampled 25 benchmark examples 100 times; Figure 2 shows GPT-4 data points.Source: Figure 2 · p. 4, §3.2 · p. 4
4 MatPlotAgent
MatPlotAgent organizes plotting as a three-part workflow: query expansion plans the task, a code agent generates and debugs code, and a visual agent critiques drafts to guide refinement.
4.1 Query Expansion
- Query expansion converts high-level requirements into explicit instructions, including which libraries and functions to use and how to prepare and manipulate data.The module acts as a planning step before figure generation.Source: §4.1 · pp. 4–5
4 MatPlotAgent
- MatPlotAgent combines query expansion, a code agent, and a visual agent in a workflow that refines figures using visual feedback.The query expansion module turns the user query into detailed instructions, which the code agent uses to generate plotting code; the visual agent then provides feedback on the draft.Source: §4 · p. 4, Figure 3 · p. 5
4.2 Code Agent
- The code agent generates plotting code from the detailed instructions and iteratively identifies and corrects bugs, with self-debugging capped at three iterations.The code agent is the framework’s core plotting component.Source: §4.2 · p. 5
4.3 Visual Agent
- The multimodal visual agent examines draft figures and sends suggestions to the code agent, incorporating visual signals that may reveal errors or weaknesses not apparent in code.Its guiding principles include checking alignment with the data and improving colors or labels for informativeness.Source: §4.2 · p. 5, §4.3 · p. 5
5 Experiments
The experiments evaluate MatPlotAgent across code and multimodal LLMs, benchmark settings, and visual-feedback analyses. The framework improves several models, including on an external benchmark, while case studies show that difficult examples remain unsolved.
5.1 Setup
- The setup compares direct decoding, zero-shot chain-of-thought, and MatPlotAgent across code LLMs, with GPT-4V and Gemini Pro Vision as visual agents.The code-model set includes GPT-4, GPT-3.5, and five open-source LLMs; the evaluation uses MatPlotBench and the proposed automatic scoring mechanism.Source: §5.1 · p. 6
5.2 Main Results
- Zero-shot chain-of-thought improves only Deepseek-coder-33B-instruct among the reported examples, from 30.88 to 36.10, while GPT-4 falls from 48.86 to 45.42.The passage reports poorer performance for other models when zero-shot chain-of-thought is applied.Source: §5.2 · p. 6
- MatPlotAgent improves GPT-4 and GPT-3.5 by 12.30 and 9.48, respectively, and Magicoder-S-DS-6.7B reaches 51.70 versus GPT-4 direct decoding’s 48.86.With Gemini Pro Vision as the visual agent, improvements over direct decoding are 7.87 and 5.45, respectively.Source: §5.2 · pp. 6–7
5.3 Results on Qwen-Agent Code Interpreter Benchmark
- On the Qwen-Agent visualization subsets, MatPlotAgent scores 72.62 versus GPT-4’s 66.7 on Visualization-Hard and 68.35 versus 60.8 on Visualization-Easy.Table 3 reports the external-benchmark comparison; without visual feedback, scores are 66.67 and 65.82.Source: Table 3 · p. 7, §5.3 · p. 7
5.4 Ablation Study
- Without visual feedback, outcomes are significantly poorer for GPT-4 and GPT-3.5; in case C, feedback resolves jumbled figure text without consulting the ground-truth figure.Table 4 reports the quantitative visual-feedback analysis; Figure 4 illustrates representative outputs from GPT-4 and Magicoder-S-DS-6.7B.Source: Table 4 · p. 7, §5.4 · p. 7, Figure 4 · p. 8
5.5 Case Study
- The case study ranges from a simple example correctly plotted by GPT-4 with MatPlotAgent to a hardest example none of three models plots correctly.On the intermediate example, GPT-4 and Magicoder-S-DS-6.7B produce drafts but omit elements; the authors characterize MatPlotBench as challenging even for GPT-4 with MatPlotAgent.Source: §5.5 · p. 7
6 Related Work
Related work covers code LLMs and LLM-agent frameworks. The paper situates its contribution in applying agentic methods to scientific data visualization.
Code LLMs
- The code-LLM discussion traces open-source models from SantaCoder and StarCoder to Code Llama, DeepSeekCoder, and Magicoder.It notes DeepSeekCoder’s 1.3B–33B range and reports that Magicoder surpasses many open-source code LLMs.Source: §6 · p. 8
LLM Agents
- The agent-framework discussion spans everyday-task platforms, simulated human behavior, Minecraft exploration, and software development.Examples include OpenAgents, Park et al. (2023), Voyager, and Chat-Dev; this study explores LLM agents for scientific data visualization.Source: §6 · p. 8
7 Conclusion
The paper introduces MatPlotBench for automated quantitative evaluation and MatPlotAgent for enhancing LLM plotting abilities. Experiments report improved performance across various LLMs.
7 Conclusion
- MatPlotBench supports automated quantitative evaluation and its scores strongly align with human assessment.The benchmark is presented as supporting evaluation of a multifaceted task requiring coding and visual skills.Source: §7 · p. 8
- MatPlotAgent uses visual feedback to enhance plotting abilities, and experimental results show improved performance across various LLMs.Source: §7 · pp. 8–9
8 Limitations
MatPlotBench targets general scientific data visualization, while the appendix describes prompts for automatic evaluation and MatPlotAgent modules.
8 Limitations
- Because scientific visualization needs vary across disciplines, the general-purpose MatPlotBench may not cover every domain-specific requirement.The authors say its applicability may therefore be restricted in certain fields and suggest domain-customized data construction and evaluation.Source: §8 · p. 9
A Detailed Prompts
- The prompt appendix introduces prompts for automatic evaluation and for MatPlotAgent’s query expansion, code agent, and visual agent.Source: A Detailed Prompts · p. 12
A.1 Evaluation Prompts
- The evaluation prompt asks GPT-4V to score a generated plot from 0 to 100 with reference to the ground-truth plot.Source: §A.1 · p. 12
A.2 Prompts for MatPlotAgent
- The query-expansion prompt requests step-by-step Python instructions, while code-agent prompts generate executable plotting code and correct bugs using interpreter errors.The visual-agent prompt asks a multimodal LLM to understand the query, analyze the draft, and provide refinement feedback.Source: §A.2 · p. 12
B Human Evaluation Details
Human annotators were recruited from computer science departments at various universities and compensated slightly above the prevailing market rate. They were told the data would be used solely for academic research and their personal information would not be disclosed.
B Human Evaluation Details
- Annotators were recruited from computer science departments at various universities and compensated slightly above the prevailing market rate.The paper says they were engaged via social media.Source: B Human Evaluation Details · p. 12
- Figure 6 presents the automatic evaluation prompt for GPT-4V.Source: Figure 6 · p. 12
- Annotators were informed that collected data would be used solely for academic research and their personal information would not be disclosed.Source: B Human Evaluation Details · p. 12
B.1 Evaluation Guide for Human Annotators
The appendix identifies prompts used for the visual agent and for query expansion, code generation, and self-debugging in MatPlotAgent. It also points to detailed instructions for human annotators scoring generated plots.
B.1 Evaluation Guide for Human Annotators
- Figure 11 gives human annotators detailed instructions for scoring model-generated plots.Source: §B.1 · p. 12
- Figure 7 presents MatPlotAgent’s query expansion prompt.Source: Figure 7 · p. 13
- Figure 8 presents MatPlotAgent’s code generation prompt.Source: Figure 8 · p. 13
- Figure 9 presents MatPlotAgent’s self-debugging prompt.Source: Figure 9 · p. 13
- Figure 10 presents the prompt for the visual agent.Source: Figure 10 · p. 13
Plot Correctness (0-100 points)
The human scoring guide defines an exact-match band for generated plots as 90–100 points. Figure 11 provides detailed instructions for annotators scoring plots.
Plot Correctness (0-100 points)
- 90–100 points marks an exact match: a generated plot is nearly identical to the ground truth, with only minor, negligible differences.Source: Plot Correctness (0-100 points) · p. 14
- Figure 11 provides human annotators with detailed instructions for scoring model-generated plots.Source: Figure 11 · p. 14