PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code

Jiaxiang Tang, Yi Zhou, Chad DeLuca, Rogerio Feris, Ahmed Khalil Omran, Zhi-Li Zhang, Pengyuan Li, Ali Anwar

arXiv:2609.38642v1cs.AIcs.CV

TL;DR

Chart editing requires mapping visual instructions to code while preserving unrelated content, but existing approaches do not jointly distinguish completed requirements, missed coupled updates, and gratuitous changes. ChartRevise builds a grammar-derived dataset and a reference-free protocol that separately checks those outcomes. Across external benchmarks, fine-tuning improves requirement recall and exact-edit rate, while preservation improves less consistently.

  • Problem

    Existing resources have limited systematic coverage across chart types and libraries, while evaluation does not jointly distinguish requirement completion, missed coupled updates, and gratuitous changes.

    Source: §1 · p. 2

  • Method

    ChartRevise builds source-grounded edits from grammar-derived templates and evaluates them with reference-free, requirement-level checks for completion, coupled updates, and preservation.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    Across five models and four external benchmarks, fine-tuning improves requirement recall in 19 of 20 comparisons and exact-edit rate in 17, but reduces gratuitous changes in only 10.

    Source: §5 · p. 11

  • Takeaways & Limitations

    Preservation on complex requests remains the main open challenge despite gains in requirement recall and exact-edit rate.

    Source: §5 · p. 11

    The code-only protocol does not directly show rendered layout, and general aesthetic quality remains outside its evaluation scope.

    Source: §4.4 · p. 11

Abstract

from arXiv · show

Chart editing requires cross-modal edit grounding, realizing a requested visual change in the code that draws it, with necessary related updates and without altering unrelated content. Existing benchmarks emphasize either code executability or chart quality, but their metrics do not clearly distinguish request completion from missed coupled updates and gratuitous changes. We introduce ChartRevise, a structured dataset and evaluation protocol for exact program-grounded chart editing. For dataset construction, we build on the grammar of graphics to systematically cover chart-editing operations, using source-program checks to verify their applicability across chart types and libraries. To improve edit exactness, our pipeline checks individual requirements and guides repair or exclusion when they are unmet. The resulting dataset contains 92,438 records covering 344 edit types across 20 chart types and three plotting libraries. For evaluation, our reference-free protocol separately measures atomic requirement completion, identifies gratuitous changes, and detects missed coupled updates. These checks are combined with successful execution and rendering to determine exact-edit success. Across five models and four external benchmarks, fine-tuning yields relative gains of 16\% in mean requirement recall and 22\% in mean exact-edit rate.

1 Introduction

Program-grounded chart editing must connect visual instructions to source-code regions while preserving unrelated chart properties, yet existing resources and metrics leave coverage and exactness gaps. ChartRevise combines grammar-derived edits and requirement-level construction audits with reference-free checks for completion, coupled updates, and gratuitous changes.

1 Introduction

  • Under-editing can omit a requested component or its dependent update, while over-editing changes unrelated content; existing resources and metrics do not jointly distinguish these errors.The motivating example’s over-edited reference completes the request but also rewrites the title.Source: §1 · p. 2
  • Fine-tuning improves requirement recall and coupled-update recall in 19 of 20 model–benchmark comparisons and exact-edit rate in 17, but reduces gratuitous changes in only 10.The transfer analysis spans five models and four external benchmarks.Source: §1 · p. 3
  • The release contains 92,438 records spanning 344 edit types, 20 chart types, and three plotting libraries, including compound requests of up to twelve actions.A human audit finds 84% accepted-set precision for the construction pipeline.Source: §1 · p. 3
  • Construction checks completion and preservation, audits coupled updates and compound requests requirement by requirement, and uses failed checks to guide repair or exclusion.The evaluation protocol separately scores explicit requirements, coupled updates, and preservation without a reference edit.Source: §1 · p. 3

2 Related Works

Prior chart-editing resources cover varied interaction formats, but their operation inventories and evaluation checks leave gaps in systematic cross-library coverage and exact-edit assessment. ChartRevise’s framing combines grammar-derived edits with joint checks of completion, coupled updates, and preservation.

Chart-editing resources

  • Editing resources cover varied interaction formats, but their inventories typically rely on predefined chart-object edits or library-specific APIs, limiting systematic extension across libraries.The related-work discussion contrasts those inventories with shared grammar-derived object–action templates.Source: §2 · p. 3

Evaluation

  • Existing benchmarks report aggregate quality or partial checks, which can miss individual omissions, unrequested changes, or necessary updates beyond the checked conditions.Assertions and text-to-geometry checks cover parts of the task but do not establish all requirements and dependent updates.Source: §2 · pp. 3–4
  • ChartRevise jointly checks completion, coupled updates, and gratuitous changes in source and edited programs, including programs that fail to execute.The comparison is framed around task scope, construction checks, and explicit evaluation checks.Source: §2 · pp. 3–4, Table 1 · p. 4

3 Systematic Construction and Validation of ChartRevise

ChartRevise constructs verified examples for cross-modal chart-edit grounding by combining grammar-derived operations, source-program checks, and staged validation. Its pipeline audits requested changes and dependent updates while preserving unrelated chart properties.

3 Systematic Construction and Validation of ChartRevise

  • An edit is exact when its program executes and renders, meets explicit requirements and coupled updates, and makes no gratuitous change.The construction pipeline uses source-grounded instructions and multi-stage validation to produce verified editing examples.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Grammar-derived edit taxonomy

  • Twelve edit categories derive from the grammar of graphics, its layered formulation, and view composition [Wilkinson, 2011, Wickham, 2010, Satyanarayan et al., 2016, 2015].Templates pair editable objects with applicable actions and can be reused across compatible chart types and plotting libraries.Source: §3.1 · p. 5

3.2 Applicability, source grounding, and composition

  • Source-program checks filter templates by chart type and verify that each target object or plotting context exists, producing 344 applicable edit types.Compatible combinations are screened before source-specific binding; compound requests combine 4–6 actions or 8–12 actions and check for direct conflicts.Source: §3.2 · p. 5

3.3 Edited program generation, validation, and repair

  • Candidate programs are executed and rendered, then checked against before/after renders, code diffs, and the edited program; validation failures feed repair and revalidation.Execution failures return tracebacks, while unapplied edits return the judge’s rationale.Source: §3.3 · p. 6

3.4 Post-processing audits

  • Among compound records with an initial judgment, 34.6% were marked applied despite at least one unperformed operation; atomic checks guide repair or exclusion.The audit checks consistency and preservation across the union of atomic requirements.Source: §3.4 · p. 6

3.5 Released dataset and quality assessment

  • 90,589 training and 1,849 test records span chart types and all three plotting libraries; ChartRevise covers all 12 categories versus other resources’ 4–11.Human audit estimates release-weighted task-level correctness at 84.2% and requirement-level correctness at 95.6% over 1,071 requirements in 200 audited compound tasks.Source: §3.5 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 7

4 Evaluation

Fine-tuning improves requirement completion, coupled-update consistency, and exact-edit success on external benchmarks, while preservation gains are less consistent. The evaluation protocol separates these outcomes and examines where edits remain inexact.

4.1 Reference-free evaluation protocol

  • The protocol checks explicit requirements, coupled updates, and gratuitous changes separately, then requires successful execution and rendering for exact-edit success.It also measures requirement satisfaction and other edit properties in programs that fail to run.Source: §4.1 · p. 8, §4.3 · p. 10

4.2 Main transfer results

  • Fine-tuning improves requirement and coupled-update recall in 19 of 20 comparisons and exact-edit rate in 17, but reduces gratuitous changes in only 10.The decomposition exposes preservation regressions that an overall exact-edit score alone would obscure.Source: §4.2 · p. 8
  • Approximately 15.6% in requirement recall, 19.8% in coupled-update recall, and 22.4% in exact-edit rate are the mean relative gains from fine-tuning.These means span 20 model–benchmark comparisons.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Diagnosing exact-edit performance

  • CodeScore improves in 19 of 20 comparisons by a mean 8.1 points, while ExecRate improves in five and falls by 3.0 percentage points on average.On Chart2Code-L2, CodeScore increases for every model while ExecRate falls for four.Source: §4.3 · p. 9, Table 3 · p. 10
  • After fine-tuning, all five models produce more executable outputs with full completion and fewer with zero completion, but four produce fewer executable outputs overall.Even tasks satisfying all explicit requirements may still contain coupled or gratuitous changes.Source: §4.3 · pp. 9–10, Figure 5 · p. 10
  • Preservation remains difficult for complex requests: on Chart2Code-L2, four of five models introduce more gratuitous changes after fine-tuning.For Phi-3.5-Vision, the advantage in reducing gratuitous changes reverses at 15 or more requirements, where exact-edit success stays near zero.Source: §4.3 · pp. 10–11, Figure 6 · p. 10

4.4 Reliability of the evaluation

  • The protocol and human evaluations agree on 93% of audited outputs and estimate aggregated gains of 14.0 and 14.5 percentage points, respectively.The code-only protocol can miss render-time layout effects such as overlapping elements; general aesthetic quality is outside its scope.Source: §4.4 · p. 11

5 Conclusion

ChartRevise combines a grammar-derived dataset with a reference-free protocol for exact program-grounded chart editing. Fine-tuning improves completion and exact-edit outcomes on external benchmarks, while preservation on complex requests remains an open challenge.

5 Conclusion

  • ChartRevise provides a grammar-derived dataset and reference-free protocol, and its protocol agrees with human exact-edit judgments on 93% of audited outputs.Across five models and four external benchmarks, fine-tuning improves requirement recall in 19 of 20 comparisons and exact-edit rate in 17, but reduces gratuitous changes in only 10.Source: §5 · p. 11

AI Use Statement

  • Generative AI assisted program generation, validation, repair, auditing, evaluation judging, and manuscript editing; the authors report reviewing all AI-assisted work.Dataset quality checks included execution, cross-model audits, and human validation of sampled records.Source: AI Use Statement · p. 12

Ethics Statement

  • The dataset uses publicly available synthetic chart programs, while the four external benchmarks are used only for evaluation and are not redistributed.The charts contain no personal information, and the authors advise verifying generated charts before use.Source: Ethics Statement · p. 12

Reproducibility Statement

  • The appendix documents construction, training, evaluation prompts, diagnostics, release schema, and human-audit criteria; the authors plan to release the dataset, code, and prompts upon publication.Source: Reproducibility Statement · p. 12

A Extended Related Work

  • Related work spans chart editing, multimodal instructions, image-space edits, and evaluation; ChartRevise emphasizes source-grounded applicability and jointly checks completion, coupled updates, and unrelated changes.Its evaluation requires successful execution and rendering for exact-edit success.Source: A Extended Related Work · p. 17

B Dataset Construction and Prompts

The appendix details how ChartRevise defines and extends its operation inventory, grounds instructions in source programs, validates and repairs generated edits, and audits compound records.

B Dataset Construction and Prompts

B.1 Taxonomy scope and library realizations

  • Edit categories are realized through program objects, and extending coverage requires new targets, action templates, applicability checks, or chart–library compatibility entries.Twelve augmentation templates extend existing categories under the same requirements.Source: §B.1 · p. 17

B.2 Operation inventory

  • The inventory contains 344 object–action templates across twelve categories, with target objects and released record counts documented in separate tables.Source: §B.2 · p. 17, Table 4 · p. 18

B.3 Grounded instruction materialization

  • Instruction placeholders take values from source programs, while some arguments are grounded in rendered geometry, such as figure width or legend anchoring.Source: §B.3 · p. 18, Table 5 · p. 18

B.4 Program generation: editor prompt

  • The editor receives the instruction and full source program and returns a complete-file rewrite, explicitly preserving unrelated chart content unless requested otherwise.The prompt requires a complete runnable program rather than a patch.Source: §B.4 · p. 19

B.5 Initial collection validation

  • Initial collection validation uses the instruction, target, programs, diff, and before/after renders to judge whether the edit reached the named object and whether unrelated objects or properties changed.The four independent flags cover application, additional changes, position preservation, and object removal.Source: §B.5 · p. 19

B.6 Feedback repair prompt

  • Repair is attempted when an edit fails to render or is judged unapplied, using the traceback or reviewer feedback while retaining the original instruction.The release was cleaned in three passes, each allowing up to two further attempts.Source: §B.6 · p. 20

B.7 Compound auditing and post-processing

  • The compound-record audit recovered 472 of 710 retried defective records, excluded unstable and other records, and left 4,519 compound records; defective instructions were removed.The reported omission rate was 34.6% among 4,611 tier-2 and tier-3 records with an initial verdict.Source: §B.7 · p. 20

C Dataset Characterization and Release Validation

The appendix describes dataset coverage and released composition, while cautioning that populated compatibility cells do not imply uniform record counts or attainment of collection targets.

C.1 Coverage notes

  • Coverage notes describe how missing source objects are supplied by deriving an auxiliary chart and checking the required precondition again.Source: §C.1 · p. 20
  • Of 4,619 populated released template–chart pairs, 677 contain at least 25 records; coverage is non-uniform, and populated cells do not guarantee reaching the collection target.The initial matrix had 4,316 eligible cells, with 25 successful records per cell as a sampling target, not a guarantee.Source: §C.1 · p. 21

C.2 Released dataset composition

  • The released composition table reports final training and test records by category, listing compound requests separately; the dataset spans 20 named chart types.Source: §C.2 · p. 21, Table 6 · p. 21

C.3 Dataset characterization and human assessment

The section describes a human audit of accepted training records and the criteria used to assess record correctness. It also characterizes strict success across categories and collection subsets.

C.3 Dataset characterization and human assessment

  • The audit executes and inspects 500 accepted records: 300 single-action and 200 compound records.The records are checked against their source.Source: §C.3 · p. 21
  • Task-level correctness requires successful execution, satisfaction of every explicit requirement, and preservation of unrelated chart content.The training-set estimate weights the two audited subsets by their proportions; requirement-level correctness is assessed within the compound-record sample and does not require the whole record to pass.Source: §C.3 · p. 21
  • Figure 7 separates the applied/strict success gap from strict success by collection subset; tiers 2 and 3 contain 4–6 and 8–12 actions, respectively.The dotted lines in panel (a) mark earlier export means.Source: §C.3 · p. 21, Figure 7 · p. 22

D Training and Evaluation Details

D.1 Training configuration

All five models are full-parameter supervised fine-tuned for one epoch on the released training split, with shared optimization and precision settings.

D.1 Training configuration

  • All five checkpoints use full-parameter supervised fine-tuning for one epoch on 90,589 training records, starting from the corresponding public base checkpoint.Shared settings include learning rate 1 × 10−5 with a cosine schedule, bfloat16 precision, weight decay 0, gradient clipping at 1.0, and gradient checkpointing; evaluation uses the end-of-epoch weights.Source: §D.1 · p. 22
  • The per-model settings table lists configurations that differ across model families; effective batch is per-device batch × gradient accumulation × number of GPUs.Source: §D.1 · p. 22, Table 8 · p. 23
  • 843 records exceeding the 6,144-token limit are dropped rather than truncated.Source: §D.1 · p. 23

D.2 Inference and evaluation configuration

The exact-edit protocol judges code against atomic requirements and necessary coupled updates, while separately checking gratuitous changes and execution. Its staged prompts establish requirements and dependencies before auditing the edited program.

D.2 Inference and evaluation configuration

  • Editing models receive the rendered source chart, source program, and instruction; the code-only judge receives program and task text, while execution and rendering are checked separately.Evaluation runs generated programs in a shared rendering environment with a 60-second timeout.Source: §D.2 · p. 23, §D.3 · p. 23

D.3 Exact-edit protocol prompts

  • Exactness distinguishes unmet required end states from changes outside the scope of requirements and necessary companions; coupled-update counts combine omissions and incorrect implementations.The coupled-update count does not distinguish those two subtypes and should be read alongside explicit requirement completion.Source: §D.3 · pp. 23–24

D.3.1 Stage A1: requirement decomposition

  • Stage A1 splits the instruction into independently checkable visible chart changes, preserving its wording and excluding properties that are not rendered.Source: §D.3.1 · p. 24

D.3.2 Stage A2: coupled-update inventory

  • Stage A2 inventories visible source-dependent changes forced by the requirements, such as a legend or axis range that must follow changed data.The inventory names the source element and forcing requirement, including updates that happen automatically.Source: §D.3.2 · p. 25

D.3.3 Requirement satisfaction and gratuitous changes

  • The requirement audit checks each requested state and classifies additional chart-affecting changes as necessary or gratuitous, ignoring code-only differences that cannot affect the chart.Source: §D.3.3 · p. 25

D.3.4 Coupled-update satisfaction

  • The coupled-update audit records whether the forcing requirement was made and whether the edited program reaches the required end state, including through automatic synchronization.An omitted or incorrect implementation is unsatisfied; if the trigger was not made, the update is also marked unsatisfied.Source: §D.3.4 · p. 25

D.4 Examples of requirements, coupled updates, and gratuitous changes

  • Examples distinguish a stale legend as a missed coupled update, an unrelated value rewrite as gratuitous, and an omitted requested color change as partial completion.A width change can satisfy one requirement while leaving a separate color requirement unmet.Source: §D.4 · pp. 26–28

E Supplementary Evaluation Results

E.1 Complete benchmark-facing transfer results

Fine-tuning improves completion measures in most benchmark comparisons, while benchmark-facing scores show less consistent patterns across metrics. The supplementary sections also describe protocol diagnostics, human validation, exactness filtering, and the released record schema.

E.1 Complete benchmark-facing transfer results

  • Both completion measures improve in 19 of 20 comparisons; CodeScore improves in nearly every comparison, while ChartScore is mixed and ExecRate is least stable.CodeScore, ChartScore, and ExecRate are not task-level success rates; the latter two use benchmark-specific scoring subsets or capture execution, respectively.Source: §E.1 · p. 29

E.2 Protocol diagnostics

  • ChartSync native text, geometry, and whole-output scores are treated as complementary diagnostics, while terminal exception labels describe execution failures.Source: §E.2 · p. 29, Table 12 · p. 30

E.3 Reference-program calibration

  • ChartM3 references have an AppliedRate of 0.345 despite an execution rate of 0.91 and ChartScore of 89.26; these aggregate scores do not isolate reference errors from adaptation or judging errors.AppliedRate checks whether an edit reaches its named target, whereas ChartScore assesses overall rendered-chart quality.Source: §E.3 · p. 30, Table 13 · p. 31

E.4 Human validation of exact-edit judgments

  • The human audit compares 200 paired external-task outputs, all of which render successfully, using blinded judgments against requirements and gratuitous-change criteria.A positive human exact-edit verdict requires every listed requirement to be satisfied and no gratuitous change.Source: §E.4 · pp. 31–32, Table 14 · p. 31

E.5 Effect of exactness filtering

  • After exactness filtering, coupled-update recall rises by 3.0 and 6.9 percentage points for Phi-3.5-V and Qwen3.5-4B on ChartM3; on Chart2Code-L2, Qwen3.5-4B requirement recall and full completion rise by 2.5 and 3.3 points while exact-edit rate is stable.Source: §E.5 · p. 32, Table 15 · p. 32

F Dataset Release and Usage

F.1 Record schema

  • The collection ledger links each candidate to its source and template and stores task descriptions, timing and attempt history, preservation status, and diagnostic signals.A nontrivial code diff does not establish that the requested chart property changed, and a positive first-pass flag alone does not certify final exactness.Source: §F.1 · pp. 32–33

F.2 Datasheet notes

The collection uses ChartNet source programs, while the pipeline generates instructions and edited programs; its taxonomy coverage is non-uniform. The notes also specify the models and compute costs used for editing and judging.

F.2 Datasheet notes

  • Source programs come from ChartNet [Kondic et al., 2026], and the pipeline generates instructions and edited programs, with non-uniform taxonomy coverage.Source: §F.2 · p. 33
  • Editing and first-pass judging use locally served gemma-4-31B-it with vLLM and W4A16 quantization; cross-auditing uses Qwen3.8-27B under the same exact-edit protocol.Source: §F.2 · p. 33
  • Per-sample editor generation averages 66 s and judging averages 17 s, while rendering takes under 1 s.The full collection entails on the order of 1.8 × 10^5 editor calls and an equal number of judging operations.Source: §F.2 · p. 33

Related papers

Loading 2609.38642v1…