PaperlayerView exact revision on arXiv

Source-linked AI summary

EdiTikZ: Scientific Figure Editing from Revision Trajectories

Christian Greisinger, Zhixue Zhao, Steffen Eger

arXiv:2609.01409v1cs.AIcs.CLcs.CV

TL;DR

Scientific figure editing is important but underexplored, and existing supervision often relies on proprietary agents or synthetic edits. EdiTikZ mines natural revision trajectories to build datasets and train compact editing models, with EdiTikZ-9B-RL leading automatic evaluation and achieving strong human ratings against proprietary systems.

  • Problem

    Publication-ready scientific figures require precise iterative editing, but scientific figure editing remains underexplored and existing approaches rely on proprietary agentic systems or synthetic edit supervision.

  • Method

    The paper mines arXiv, GitHub, and TeX SE revision trajectories, infers bidirectional instructions from rendered figures and TikZ code, and trains 4B and 9B models with joint reconstruction-editing learning and multi-reward RL.

  • Results

    EdiTikZ-9B-RL leads automatic evaluation, exceeds GPT-5.6-Sol and nearly matches Gemini-3.1-Pro in human evaluation, with gains transferring to substantially more complex OOD figures.

  • Takeaways & Limitations

    Naturally occurring revision trajectories provide effective supervision for compact open scientific-figure editing models competitive with larger proprietary systems.

  • Takeaways & Limitations

    Automatically inferred instructions retain omissions and misinterpretations, while performance degrades on long OOD generations and evaluation there may be less reliable.

Abstract

from arXiv · show

Vision-language models (VLMs) have shown strong performance in generating scientific figures from text or images. However, producing publication-ready figures requires iterative refinement, making scientific figure editing an important yet largely unexplored task. Existing approaches rely on costly proprietary agentic systems, focus primarily on evaluation, or construct training supervision from synthetically generated edits. Instead, we leverage naturally occurring scientific revision and development trajectories as a scalable source of supervision. To this end, we introduce DaEdiTikZ, the first large-scale dataset of revision-derived scientific figure edits, constructed by mining 391K plausible TikZ edit pairs from arXiv, GitHub, and TeX SE and inferring 781K directed edit instructions with a VLM conditioned on rendered figures and TikZ code. We further introduce DaEdiTikZ-Bench, a human-refined benchmark with 790 instances, and train two compact Qwen3.5-based EdiTikZ models (4B and 9B) by jointly learning reconstruction and editing, followed by reinforcement learning (RL) with complementary rewards for rendered fidelity and edit application. Automatic evaluation places our 9B model above all tested baselines, while human evaluation with 9 annotators and 4,320 ratings places it above GPT-5.6-Sol and on par with Gemini-3.1-Pro. Under severe out-of-distribution shifts, it remains competitive with GPT-5.6-Sol near its 2K training sequence-length regime. Models and datasets will be released.

1 INTRODUCTION

Scientific figure editing remains underexplored despite the need for precise revisions to make generated figures publication-ready. EdiTikZ addresses this gap with revision-derived supervision, datasets, and compact models that achieve strong automatic and human-evaluated performance.

  • Publication-ready scientific figures require precise natural-language-guided revisions, but editing existing figures remains underexplored.
  • EdiTikZ mines naturally occurring revisions from arXiv, GitHub, and TeX SE to create 391K plausible TikZ edit pairs and 781K directed editing instances.
  • DaEdiTikZ-Bench provides 790 human-refined editing instances for evaluation.
  • The 4B and 9B EdiTikZ models jointly learn reconstruction and editing, then use reinforcement learning with rendered-fidelity and edit-application rewards.
  • EdiTikZ-9B-RL exceeds GPT-5.6-Sol and matches Gemini-3.1-Pro in human evaluation while leading automatic evaluation.

2 RELATED WORK

Prior work largely addresses scientific-figure generation or editing through agentic systems and synthetic supervision. EdiTikZ is positioned within a broader literature on graphics-program generation, figure editing, and rendering-feedback reinforcement learning.

  • Generating Scientific Figures with Graphics Programs: Prior scientific-figure generation work produces graphics programs from text or images across TikZ, SVG, Python, and other visualization languages.
  • Scientific Figure Editing: Scientific figure editing has been studied for charts, SVGs, TikZ, and raster images, often using agentic systems.
  • Scientific Figure Editing: Concurrent approaches include unified scientific-image understanding, generation, and editing, and OCR-based synthetic editing supervision.
  • RL from Rendering Feedback: Rendering-feedback reinforcement learning has been applied to SVG and TikZ generation using perceptual, domain-specific, code-based, and self-consistency rewards.

3 DATASET AND BENCHMARK

DaEdiTikZ is constructed by mining related scientific figures, pairing them by visual similarity, and using a VLM to infer bidirectional edit instructions. Quality filtering and human refinement produce a large training corpus and a 790-instance benchmark.

  • Revision-Derived Editing Supervision: Revision sources include cross-version figures, related subfigures, alternative TikZ programs, and iterative TeX SE refinements.
  • Collecting Scientific Revision Traces: Historical arXiv versions add 0.77M figures, expanding the unique arXiv corpus from 1.47M to 2.38M and producing 2.91M unique TikZ figures with GitHub and TeX SE.
  • Recovering Plausible Edit Pairs: 222K source groups yield 430,442 candidate pairs after grouping, pruning, and cosine-similarity filtering.
  • Inferring Edit Instructions: A VLM conditioned on rendered figures and TikZ code accepts both directions to produce 390,516 pairs and 781,032 directional editing trajectories.
  • Dataset Quality Analysis: 98% of retained transformations are plausible, and 82.9% of instructions receive good or very good quality ratings.
  • Dataset Quality Analysis: Removing TikZ-code grounding raises the error rate from 50% to 80%, with omissions increasing by 16 percentage points and numeric misinterpretations from 1% to 8.5%.
  • DaEdiTikZ-Bench: Manual inspection leaves 395 revision pairs and 790 editing instances in DaEdiTikZ-Bench after removing trivial edits and rendering artifacts.

4 EDITING-SPECIFIC POST-TRAINING

The paper jointly trains editing and reconstruction, then applies GDPO with complementary rewards for rendered similarity and instruction following. The two-stage setup uses disjoint data between SFT and RL and freezes the vision encoder during RL.

  • Joint Reconstruction and Editing SFT: 752K editing triplets are jointly trained with 752K reconstruction samples to strengthen the shared image-to-TikZ mapping.Reconstruction is conditioned only on the rendered target figure, while editing uses source figure, instruction, and target TikZ.
  • Editing-Specific Rewards: RL reuses a frozen SFT vision encoder to compute rendering-based similarity rewards for sampled TikZ rollouts.The encoder is frozen during RL to prevent reward hacking.
  • Editing-Specific Rewards: Instruction-following rewards provide proportional credit for partially applied edits, complementing target-similarity rewards.Target similarity alone can reward preserving unchanged content or penalize valid outputs when inferred instructions are imperfect.
  • Editing-Specific Rewards: The pipeline gates similarity and instruction-following rewards on compilation and standalone TikZ-format validity.Failed compilations or invalid formats receive zero reward.
  • Multi-Reward Optimization with GDPO: GDPO normalizes each reward independently before aggregation, reducing sensitivity to differences in reward distributions.The objective is optimized over grouped rollouts after advantage normalization.

5 EXPERIMENTS

Experiments use disjoint group-level splits, compare compact EdiTikZ models with proprietary and open-source VLMs, and evaluate both code and editing-specific human criteria.

  • Setup: 27K DaEdiTikZ trajectories are reserved for RL, while 754K editing trajectories and 754K reconstruction samples support SFT.Figures from the same group never occur across training stages; evaluation uses 790 human-refined benchmark instances.
  • Models: The evaluation compares six proprietary VLMs and eight open-source VLMs, with SFT applied to models up to 9B parameters.RL is applied to EdiTikZ-4B and EdiTikZ-9B.
  • Metrics: Evaluation combines TeX Edit Distance and DreamSim with Edit Application, Source Preservation, and Visual Quality criteria.The human criteria measure requested-edit correctness, unaffected-content preservation, and legibility or publication readiness.

6 RESULTS

Post-training substantially improves EdiTikZ performance, with EdiTikZ-9B-RL achieving the strongest automatic results and near-leading human ratings. Gains persist under severe distribution shifts, though performance degrades beyond the 2K-token training horizon and code similarity does not track visual correctness reliably.

  • Automatic Evaluation: EdiTikZ-9B-RL achieves the highest overall automatic score, reaching 0.726 Avg after RL.Across architectures, SFT improves Avg by 0.186–0.363 and compilation rate by 19.0–39.3 percentage points; RL reaches 0.674 and 0.726 Avg for the 4B and 9B models.
  • Automatic Evaluation: SFT reverses the initial model ranking: Qwen3.5-4B/9B rise from 0.249/0.345 to 0.612/0.643 Avg after training.Before SFT, Qwen3-VL-4B/8B score 0.314/0.354 Avg, exceeding the corresponding Qwen3.5 models; after SFT, they score 0.538/0.540.
  • Metric Alignment: Editing weakens the relationship between visual correctness and code similarity: TED worsens after RL despite consistent gains in rendered metrics.The authors hypothesize that visually equivalent edits can differ at the code level.
  • Human Evaluation: Human ratings place EdiTikZ-9B-RL above GPT-5.6-Sol and near Gemini-3.1-Pro across edit application, source preservation, and visual quality.The combined score is 17.43 for EdiTikZ-9B-RL, versus 16.75 for GPT-5.6-Sol and 17.72 for Gemini-3.1-Pro; 9 annotators provide 4,320 ratings.
  • Metric Alignment: The aggregate automatic metric correlates strongly with combined human judgments at ρ = 0.823, whereas TED correlates poorly at ρ = 0.374.DSim and criterion-specific EA, SP, and VQ metrics each reach ρ ≈0.77; adding RIF raises overall correlation from 0.812 to 0.827.
  • Ablations: Joint reconstruction and editing training improves editing, while GDPO benefits from independently normalized complementary rewards.Joint training reaches 0.432 Avg versus 0.392 for editing-only at 3B, and GDPO raises the combined-reward gain to +0.038 Avg versus +0.009 with GRPO.
  • OOD Generalization: RL gains increase with generation difficulty, dominating SFT from 1.5–4K tokens while maintaining >80% compilation through 3–4K.Within the trained ≤2K regime, EdiTikZ-9B-RL remains within <0.1 Avg of GPT-5.6-Sol; beyond 2K, the gap widens as EdiTikZ degrades faster.

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

The paper presents revision mining as effective supervision for compact scientific-figure editing models, with strong benchmark performance and transfer to complex OOD figures. It also identifies noisy inferred instructions and long-generation OOD degradation as limitations, motivating future training on more complex figures.

  • Conclusion: DaEdiTikZ mines naturally occurring revisions from arXiv, GitHub, and TeX SE, supporting 4–9B EdiTikZ models trained with multi-task SFT and multi-reward RL.The paper also introduces the human-refined DaEdiTikZ-Bench.
  • Conclusion: EdiTikZ-9B-RL leads automatic evaluation and reaches human ratings comparable to the strongest proprietary system, with gains transferring beyond the 2K-token horizon.The conclusion reports transfer to substantially more complex SPIQA and CharXiv figures.
  • Limitations: DaEdiTikZ inherits omissions and misinterpretations from automatically inferred instructions, despite filtering and code grounding.Performance also degrades for long OOD generations, while evaluation there is limited by synthetic instructions and potentially less reliable reference-free judging.

A.1.1 INFERRING EDIT INSTRUCTIONS

The dataset infers directional, atomic edit instructions from paired TikZ figures and code, then analyzes their quality, semantics, similarity, and source-specific characteristics. The resulting trajectories are predominantly modifications, with edit types and description lengths varying systematically across operations and similarity levels.

  • Inferring Edit Instructions: Qwen3.6-27B infers both edit directions from rendered figure pairs and TikZ code, classifying each as ok, invalid, or identical before decomposing accepted transformations.Each atomic edit specifies an intent, operation, and natural-language description.
  • Inferring Edit Instructions: The output schema records pair quality and atomic edits using add, remove, or modify intents across text, annotation, geometry, style, data, and structure operations.Instructions report only visible differences and require concrete before→after descriptions.
  • Dataset Statistics: 781K directional trajectories and 3.28M atomic edits connect 590K unique figures, while 69.3% of figures occur in only one pair and 97.3% in at most three.The dataset retains 90.7% of 430K candidate pairs and contains 390.5K pairs supporting supervision in both directions.
  • Dataset Statistics: 93.2% of both forward and backward responses are accepted, yielding 390.5K pairs that support supervision in both directions.Directional quality is nearly symmetric across the two directions.
  • Edit Characteristics: 76.6% of revisions modify existing content, while text is the most frequent operation at 42.1%, followed by geometry at 20.0%.Data and structural edits have the longest descriptions, averaging 32.3 and 28.4 words, whereas text edits average 17.8 words.
  • Similarity Analysis: As similarity increases, mean edit count falls from 5.29 to 2.61 while description length rises from 20.9 to 25.4 words per edit.Lower-similarity pairs contain more additions and removals, while higher-similarity pairs contain more modifications; bidirectional retention peaks at 96.6% in [0.98, 0.99).
  • Source Characteristics: ArXiv trajectories are primarily text-centered, GitHub contains more annotation and data edits, and TeX SE has fewer atomic edits but the longest instructions.TeX SE averages 2.3 atomic edits and 26.5 words per edit, compared with 4.2 and 22 respectively elsewhere.

A.1.2 DATASET QUALITY ANALYSIS

Dataset quality analysis used two student annotators, who completed an evaluation sheet; the evaluation guidelines are summarized in Table 13.

  • Two student annotators conducted the dataset quality analysis.The team comprised one master's student and one PhD student.
  • Both annotators completed the evaluation sheet shown in Figure 9.Figure 9 documents the sheet used to evaluate directional VLM-generated edit instructions.
  • The guidelines for completing the evaluation form are summarized in Table 13.

A.1.3 DAEDITIKZ-BENCH

DaEdiTikZ-Bench is a manually refined benchmark for evaluating scientific figure editing and related reconstruction. It defines visual, source-preservation, and edit-application criteria while supporting supervised training and reinforcement learning with rendering-based rewards.

  • Benchmark Construction: Six annotators manually corrected all 790 VLM-generated benchmark instructions using source and target images plus raw JSON responses.Corrections remove hallucinations, fix misinterpretations, and add omitted changes.
  • Training and Rewards: The editing prompt is used across all training stages and model evaluation, while reconstruction prompts require standalone TikZ code wrapped in a complete LaTeX document.Joint reconstruction and editing supervision is followed by RL using complementary rendered-fidelity and edit-application rewards.
  • Evaluation Criteria: The VLM-as-a-judge evaluates edit application, source preservation, and visual quality using integer scores from 0 to 10.The criteria respectively measure requested-change completion, preservation of unaffected content, and clean, legible, publication-ready rendering.
  • Automatic Evaluation: EdiTikZ-4B and 9B achieve 0.701 and 0.748 average reconstruction scores, exceeding GPT-5.6-Sol at 0.652 and Gemini-3.1-Pro at 0.655.The models improve over their base models by +0.374 and +0.430, respectively.
  • Automatic Evaluation: Editing supervision does not improve reconstruction: EdiTikZ-8B scores 0.624 versus 0.672 for DeTikZify-8B.The authors hypothesize that reconstruction supervision strengthens editing, whereas noisy edit instructions may dilute reconstruction learning.
Loading 2609.01409v1…