Source-linked AI summary
ChartReformer: Natural Language-Driven Chart Image Editing
Pengyu Yan, Mahesh Bhosale, Jay Lal, Bikhyat Adhikari, David Doermann
TL;DR
Chart images can be difficult to adapt when their source data and supplementary information are unavailable, while traditional editing may require manual work and plotting expertise. ChartReformer uses natural-language instructions to infer chart data and visual attributes, then generates edited charts; the authors report better performance than ChartLlama across edit categories on the compared measures.
Problem
Traditional chart editing can require manual intervention, plotting-software expertise, and original data tables that may be unavailable, limiting adaptation of charts to different application needs.
Source: §1 · pp. 1–2Method
Results
ChartReformer performs better than ChartLlama across all edit categories on SSIM and success rate, though the comparison did not use prompt engineering for ChartLlama.
Source: §5.2 · p. 11Takeaways & Limitations
The approach supports natural-language edits across style, layout, format, and data-centric changes directly from chart images without underlying data tables.
Source: §1 · p. 3, §7 · p. 13Edit performance depends heavily on data-extraction accuracy, and real-world instructions may be abstract or arbitrarily complex.
Source: §6 · p. 13
Abstract
from arXiv · showhide
Chart visualizations are essential for data interpretation and communication; however, most charts are only accessible in image format and lack the corresponding data tables and supplementary information, making it difficult to alter their appearance for different application scenarios. To eliminate the need for original underlying data and information to perform chart editing, we propose ChartReformer, a natural language-driven chart image editing solution that directly edits the charts from the input images with the given instruction prompts. The key in this method is that we allow the model to comprehend the chart and reason over the prompt to generate the corresponding underlying data table and visual attributes for new charts, enabling precise edits. Additionally, to generalize ChartReformer, we define and standardize various types of chart editing, covering style, layout, format, and data-centric edits. The experiments show promising results for the natural language-driven chart image editing.
1 Introduction
ChartReformer edits chart images from natural-language prompts without requiring original data or plot settings, predicting data and visual attributes for a replotter. The work also standardizes chart-edit categories and introduces ChartCraft training data.
1 Introduction
- Traditional chart editing often requires manual intervention, plotting-software expertise, and original data tables, which may be unavailable.Chart appearances can also be suited to a specific scenario, limiting adaptation to other applications.Source: §1 · pp. 1–2
- ChartReformer edits chart images from natural-language prompts without original data or plot settings by predicting underlying data and visual attributes for a replotter.The approach decomposes the chart and reasons over the prompt to generate the parameters for a new chart.Source: §1 · p. 3
- The authors define and standardize chart-edit categories and design evaluation metrics for the task.The four categories are style, layout, format, and data-centric edits.Source: Figure 1 · p. 2, §1 · p. 3
- ChartCraft contains 100K pairs of original and edited chart images with corresponding data tables, visual attributes, and instruction prompts.Source: §1 · p. 3
2 Related Work
Prior natural-language visualization tools rely on underlying tabular data, while chart editing requires understanding both visual features and represented data. Existing chart datasets and models address related comprehension tasks but leave chart editing capabilities limited.
2.1 Natural-language-driven visualization
- Natural-language visualization tools such as VegaLite and ChartDialogs adjust charts using available underlying tabular data rather than directly editing chart images.VizGpt is also described as relying on underlying data.Source: §2.1 · p. 3
2.2 Chart Comprehension
- Chart editing requires understanding both a chart’s visual features and its data, while chart comprehension models face challenges extracting that data.Existing datasets address question answering, templates, or chart-to-text summarization; the authors report no publicly available chart-editing dataset.Source: Datasets · pp. 3–4
Datasets
- Prior models de-render charts into tables or plot code, but the authors say none explicitly predicts and adjusts both visual attributes and underlying data for editing.Predicting visualization code also carries a risk of compilation failure.Source: Datasets · p. 4
3 Problem Statement
The paper groups chart-image edits into four classes and notes that advanced edits can be formed by chaining them. The classes distinguish changes to appearance, layout, chart format, and underlying data.
3 Problem Statement
- Chart-image edits fall into style, layout, format, and data-centric classes, and advanced edits can chain these categories.Source: §3.1 · p. 4
- Style edits change low-level visual attributes without changing data or chart format; layout edits adjust axes grids and legends.Style attributes include colors, line or marker styles, bar patterns, and fonts.Source: §3.1 · pp. 4–5
- Format conversion changes chart type to present data from different perspectives, such as trends in line charts or group comparisons in bar charts.Grouped bars emphasize absolute values, while stacked bars reflect ratios.Source: §3.1 · p. 5
- Data-centric edits manipulate chart data through operations such as range or series filtering and adding data series or points.Source: §3.1 · p. 5
4 ChartReformer
ChartReformer de-renders chart images into data and visual attributes, then uses natural-language prompts to create edited charts. Its dataset pairs chart images, edit prompts, underlying data, and visual attributes across four edit types.
4 ChartReformer
- ChartReformer de-renders chart images into underlying data and visual attributes that replotting software can use to construct edited charts.The model reasons over the edit prompt to predict the corresponding data and attributes.Source: §4 · p. 6
4.1 Dataset
- The dataset uses existing chart tables, prioritizing line and bar charts and their grouped and stacked vertical or horizontal variations.The tables come from AdobeSynth-19, whose synthetic images use data derived from real-world sources.Source: §4.1 · pp. 6–7, Data Source · p. 7
Data Source
- A custom Matplotlib tool samples visual attributes and stores plotting parameters in modifiable JSON.The parameter pool supplies varied chart appearances and supports subsequent adjustments.Source: Chart Image Generation · p. 7
Chart Image Generation
- Each synthesized sample pairs edited charts with prompts, and each chart includes visual attributes, a data table, and an image.The pairs are generated from chart images.Source: Edit Pair Synthesis · p. 7
Edit Pair Synthesis
- The four edit types are style and layout changes, line-bar format conversions, and data edits that update the table while preserving visual attributes.Style and layout edits modify plotting parameters; format conversions retain the data and corresponding series colors.Source: Edit Pair Synthesis · p. 7
- Each edit pair receives 5–7 prompt variants generated from one base prompt to represent diverse natural-language expressions.Source: Edit Pair Synthesis · p. 7
- The dataset contains approximately 100K paired samples, with significantly more samples for data-centric edits because of the task challenges.Table 2 reports sample counts by edit type and chart category.Source: Dataset Statistics · p. 8, Table 2 · p. 8
Dataset Statistics
The pipeline first trains the model to recover chart data and visual attributes, then fine-tunes it to interpret edit prompts. Dataset synthesis also oversamples chart-type-relevant visual attributes.
Dataset Statistics
- Chart-type-relevant visual attributes are oversampled relative to chart-type-irrelevant attributes to emphasize the more challenging graph identification and manipulation.Source: Dataset Statistics · p. 8
4.2 Our Pipeline
- Training has two stages: pre-training for chart de-rendering followed by fine-tuning for chart editing.Source: §4.2 · p. 8
Chart De-rendering
- Pre-training on 100K unpaired chart images predicts both visual attributes and underlying data to learn their mapping.The model uses 800 × 800 padded inputs and an output limit of 1024 tokens.Source: Chart De-rendering · p. 9
Chart Editing
- Fine-tuning uses 88K paired images and edit prompts, after which predicted data and plotting parameters are used to replot edited charts.JSON repair and default plotting parameters are suggested for incomplete predictions, but not used in evaluation.Source: Chart Editing · p. 9
5 Experiments
On the reported comparisons, ChartReformer performs better than ChartLlama across edit categories on SSIM and success rate, while its own results vary by edit type. The evaluation combines image-level and edit-correctness measures.
5 Experiments
- Evaluation is conducted exclusively on the authors’ dataset because no publicly available chart-editing dataset was identified.Source: §5 · p. 9
5.1 Metrics
- Evaluation uses SSIM for image similarity and success rate for successful chart generation, with plotting failures counted as unsuccessful.For ChartLlama, success rate measures whether predicted code compiles.Source: §5.1 · p. 9
5.2 Results
- On 5.5K test samples, data-centric edits are hardest and format edits easiest, while ChartReformer maintains high image fidelity.The paper attributes the difficulty of data-centric edits to the need to understand existing data and manipulate it according to the prompt.Source: Table 3 · p. 10, §5.2 · p. 11
Evaluating Edit Correctness
- Edit correctness is measured with RMS for underlying data and VAES for visual attributes, compared against the corresponding ground truth.The replotting process is heuristic.Source: Evaluating Edit Correctness · pp. 9–10
- VAES separates attributes that should change from those that should remain unchanged and combines their scores with a harmonic mean.Categorical values use exact matches, while numeric values receive scores from 0 to 1 using a 0.4 threshold.Source: Evaluating Edit Correctness · p. 10
5.2 Results
- ChartReformer performs better than ChartLlama across all edit categories on SSIM and success rate in a 550-sample subset comparison.The authors note that they did not perform prompt engineering for ChartLlama.Source: Table 4 · p. 10, §5.2 · p. 11
6 Discussion and Limitations
The discussion identifies data extraction accuracy and instruction complexity as constraints on chart-editing performance.
6 Discussion and Limitations
- Overall edit performance depends heavily on data extraction accuracy, making more accurate extraction a proposed research direction.Source: §6 · p. 13
- Real-world instructions may be abstract and arbitrarily complex, beyond the range of the chart-edit dataset.Source: §6 · p. 13
- A proposed preprocessing module could simplify complex instructions into chained prompts that ChartReformer can handle.The authors give a decoder-only language model as one possible module.Source: §6 · p. 13
7 Conclusion
ChartReformer standardizes chart editing and introduces ChartCraft, a dataset for the task. It edits chart images through decomposed data and visual attributes across four edit categories.
7 Conclusion
- ChartReformer enables natural-language edits directly from chart images, without underlying data tables or supplementary information.Its edited-chart representation includes both a data table and visual attributes.Source: §7 · p. 13
- The standardized editing scope covers style, layout, format, and data-centric modifications.Source: §7 · p. 13
- The experiments demonstrate promising results and highlight potential to enhance chart accessibility and adaptability for diverse applications.Source: §7 · p. 13
8 Appendix
The dataset-generation process samples Matplotlib parameters to produce more diverse charts than relying on default selections.
8 Appendix
- Matplotlib parameters are randomly sampled to generate more diverse charts than using the library’s defaults.Source: Matplotlib visual attributes pool · p. 16
Matplotlib visual attributes pool
The JSON representation combines chart data and visual attributes; the Matplotlib property pool is identified as Table 5.
Matplotlib visual attributes pool
- Table 5 is identified as the Matplotlib property pool.Source: Table 5 · p. 16
- The JSON configuration includes plotting parameters and an underlying data table, associating data series with visual attributes.Source: Example of JSON Properties · p. 17
- The data table is placed before visual attributes because text generation is described as more stable at the front, while data is considered more sensitive and valuable.Source: Example of JSON Properties · p. 17
Example of JSON Properties
The line-chart JSON pairs an underlying data table with chart and plotting properties. Its fields include chart type, titles, and formatting parameters.
Example of JSON Properties
- The example JSON includes an underlying data table alongside plotting parameters, linking data series to their visual attributes.Source: Figure 6 · p. 18
- The example table lists country names and values for 2004 and 1997, including OECD members, Middle East & North Africa, and South Asia.Source: Example of JSON Properties · p. 18
- The configuration identifies a line chart and supplies chart, x-axis, and y-axis titles.Source: Example of JSON Properties · p. 18
- Formatting fields cover axis-label fonts and sizes, legend placement and columns, title formatting, tick settings, and grid parameters.Source: Example of JSON Properties · p. 18