Source-linked AI summary
PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback
Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt
TL;DR
Novice visualization users face tool-selection and plotting challenges, while code-generating LLMs often require iterative correction. PlotGen coordinates planning, code generation, and multimodal feedback agents, outperforming strong baselines on MatPlotBench and improving reported trust and debugging productivity.
Problem
Novice users struggle to select tools and master techniques for accurate scientific visualization, while LLM-generated code can fail to follow specifications.
Source: §1 · p. 1Method
PlotGen orchestrates query planning, code generation, and numeric, lexical, and visual multimodal feedback agents for iterative plot refinement.
Source: §1 · pp. 1–2, §3 · p. 3Results
4-6% improvement on MatPlotBench shows PlotGen outperforming strong baselines across varied LLM configurations.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Scientific data visualization is pivotal for transforming raw data into comprehensible visual representations, enabling pattern recognition, forecasting, and the presentation of data-driven insights. However, novice users often face difficulties due to the complexity of selecting appropriate tools and mastering visualization techniques. Large Language Models (LLMs) have recently demonstrated potential in assisting code generation, though they struggle with accuracy and require iterative debugging. In this paper, we propose PlotGen, a novel multi-agent framework aimed at automating the creation of precise scientific visualizations. PlotGen orchestrates multiple LLM-based agents, including a Query Planning Agent that breaks down complex user requests into executable steps, a Code Generation Agent that converts pseudocode into executable Python code, and three retrieval feedback agents - a Numeric Feedback Agent, a Lexical Feedback Agent, and a Visual Feedback Agent - that leverage multimodal LLMs to iteratively refine the data accuracy, textual labels, and visual correctness of generated plots via self-reflection. Extensive experiments show that PlotGen outperforms strong baselines, achieving a 4-6 percent improvement on the MatPlotBench dataset, leading to enhanced user trust in LLM-generated visualizations and improved novice productivity due to a reduction in debugging time needed for plot errors.
1 Introduction
PlotGen addresses novice difficulties in producing accurate scientific visualizations by coordinating planning, code generation, and multimodal self-reflection. It reports improved benchmark performance and user-oriented benefits.
1 Introduction
- Novice users struggle with requirements interpretation, tool selection, and plotting expertise when creating accurate visualizations from raw data.Source: §1 · p. 1
- LLMs can generate visualization code but often fail to follow user specifications faithfully, making iterative refinement necessary.Source: §1 · p. 1
- PlotGen coordinates query planning, code generation, and numeric, lexical, and visual feedback agents to iteratively refine scientific plots.The feedback agents target data accuracy, textual labels, and visual aesthetics through multimodal self-reflection.Source: §1 · pp. 1–2
- 4-6% improvement on MatPlotBench shows PlotGen outperforming strong baselines across varied LLM configurations.Source: §1 · p. 2
2 Related Work
Prior work established LLM-based chart generation, multimodal plotting, human feedback, and visual-feedback refinement, while PlotGen combines multimodal self-reflection across numeric, lexical, and visual errors.
2 Related Work
- Existing visualization systems progressed from LSTM and LLM code generation to multimodal plotting and feedback-based refinement.Source: §2 · p. 2
- PlotGen differs by using multimodal LLM self-reflection to address numeric values, lexical labels, and visual aesthetics together.Source: §2 · p. 2
- LLM-agent research spans practical platforms, behavioral simulations, autonomous game agents, and software-development systems.Source: §2 · pp. 2–3
3 Methodology
PlotGen maps a user request and data table to a scientific figure through planning, code generation, bounded debugging, and sequential multimodal feedback.
3 Methodology
- PlotGen takes visualization requirements and a data table as input and produces a scientific figure through five coordinated agents.Source: §3 · p. 3
- The Query Planning Agent decomposes complex requests into executable steps specifying code operations, data formats, and desired visual characteristics.Source: §3 · p. 3
- The Code Generation Agent converts planned instructions into executable Python and uses bounded debugger feedback to correct syntax, imports, arguments, and formatting.Source: §3 · p. 3
- Numeric, lexical, and visual agents compare or inspect draft plots and sequentially send feedback to correct data trends, labels, plot types, and aesthetics.Each feedback agent repeats until its domain is satisfactory or the maximum trial count is reached.Source: §3 · p. 3
4 Experiments
The experiments evaluate PlotGen on MatPlotBench with multiple code-LLM backbones and compare it against direct decoding, zero-shot chain-of-thought, and MatPlotAgent.
4 Experiments
- MatPlotBench contains 100 user queries paired with raw data and ground-truth visualizations for evaluating generated plots.Source: §4 · p. 3
- The evaluation uses GPT-4V LLM-as-a-judge scores from 0 to 100 and tests closed- and open-source code LLMs.Source: §4 · p. 3
- PlotGen is compared with direct decoding, zero-shot chain-of-thought, and MatPlotAgent baselines on MatPlotBench.Source: §4 · p. 3, Table 1 · p. 4
5 Results and Discussion
PlotGen outperforms strong baselines across open- and closed-source code LLMs, while multimodal feedback addresses concurrent plotting errors. Ablations and user evaluation identify visual feedback, lexical accuracy, and reduced debugging effort as important practical factors.
Performance Evaluation
- PlotGen significantly outperforms strong baselines across open-source and closed-source code LLMs on MatPlotBench.With GPT-4, combining lexical, numerical, and visual feedback helps recover from concurrent plotting errors.Source: §5 · p. 4
Ablation Study
- Removing Lexical Feedback causes a 5-7% performance deterioration because textual details such as titles, tick labels, and legends remain insufficiently refined.Source: §5 · p. 4
- Removing Visual Feedback causes a 10-15% performance drop across all code LLMs, exposing errors in colors, scaling, layout, tick labels, and legend mapping.Source: §5 · p. 4
User Evaluation
- In a study of 200 requests, participants rated 40.5% of PlotGen attributions Completely Accurate and 24.5% Somewhat Accurate, while outputs reduced debugging time.Participants rated 10.6% of cases Completely Inaccurate.Source: §5 · p. 4
6 Conclusion
PlotGen coordinates query planning, code generation, and multimodal feedback to automate visualization according to user preferences. The paper reports improved plot quality, user trust, accessibility, and debugging efficiency, while outlining extensions beyond traditional table visualization.
6 Conclusion
- PlotGen coordinates query planning, code generation, and multimodal feedback through self-reflection to generate visualizations according to user preferences.Source: §6 · p. 4
- PlotGen improves LLM-generated plot quality by 10-12% compared with strong baselines while enhancing user trust, reducing debugging time, and improving accessibility.Source: §6 · p. 4
- Future work will extend PlotGen beyond traditional table visualization to interactive dashboards, Virtual Reality simulations, and visual arts.Source: §6 · p. 4