PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning

Juekai Lin, Yun Zhu, Honglin Lin, Sijing Li, Tianwei Lin, Zheng Liu, Xiaoyang Wang, Wenqiao Zhang, Lijun Wu

arXiv:2604.06079v1cs.CVcs.AI

TL;DR

Graphics program synthesis must recover editable TikZ code despite strict spatial requirements, while existing data and benchmarks provide insufficient executability, alignment, and coverage. The paper addresses these gaps with curated data, multifaceted evaluation, and Dual Self-Consistency reinforcement learning, and reports state-of-the-art performance for SciTikZer-8B against proprietary, larger, and specialized models.

  • Problem

    Existing image-TikZ resources have limited executability and visual alignment, while evaluation lacks comprehensive coverage of visual fidelity and structural logic.

  • Method

    The paper builds SciTikZ-230K and SciTikZ-Bench and introduces Dual Self-Consistency reinforcement learning with round-trip reconstruction and verification.

  • Results

    SciTikZ-8B achieves state-of-the-art performance, including a 97.2% compilation success rate on SciTikZ-Bench versus 88.9% for Gemini-2.5-Pro.

  • Takeaways & Limitations

    The framework provides a foundation for visually grounded TikZ synthesis by jointly evaluating and optimizing compilability, visual fidelity, and structural logic.

  • Takeaways & Limitations

    Dual Self-Consistency reinforcement learning adds computational overhead through multiple forward passes and external LaTeX rendering during training.

Abstract

from arXiv · show

Graphics Program Synthesis is pivotal for interpreting and editing visual data, effectively facilitating the reverse-engineering of static visuals into editable TikZ code. While TikZ is the de facto standard for scientific schematics due to its programmatic flexibility, its requirement for rigorous spatial precision presents a significant challenge for Multimodal Large Language Models. Progress is currently stifled by two primary gaps: (1) Data Quality Gap: existing image-TikZ corpora often lack strict executability and reliable visual alignment; (2) Evaluation Gap: a lack of benchmarks for both structural and visual fidelity. To address these, we present a closed-loop framework featuring: SciTikZ-230K, a large-scale, high-quality dataset from our Execution-Centric Data Engine covering 11 diverse scientific disciplines; SciTikZ-Bench, a multifaceted benchmark spanning from basic geometric constructs to intricate hierarchical schematics to evaluate both visual fidelity and structural logic. To further broaden the scope of visual-code optimization methodology, we introduce a novel Dual Self-Consistency Reinforcement Learning optimization paradigm, which utilizes Round-Trip Verification to penalize degenerate code and boost overall self-consistency. Empowered by these, our trained model SciTikZer-8B achieves state-of-the-art performance, consistently outperforming proprietary giants like Gemini-2.5-Pro and massive models like Qwen3-VL-235B-A22B-Instruct.

1 Introduction

Graphics program synthesis converts static visuals into editable TikZ code, but TikZ’s strict spatial and symbolic requirements make errors consequential. The paper addresses data and evaluation gaps with a closed-loop framework combining curated data, comprehensive benchmarking, and Dual Self-Consistency reinforcement learning.

  • Motivation: TikZ synthesis requires precise coordinates, explicit primitives, and rigorous spatial definitions for fine-grained scientific structures.Even small errors can trigger compilation failures or structurally degenerate artifacts.
  • Research gaps: Existing corpora suffer from noisy or low-quality training signals, while benchmarks remain narrow and inadequately assess complex multidisciplinary graphics.These gaps limit reliable training and standardized evaluation of visual and structural fidelity.
  • Framework: The framework introduces SciTikZ-230K and SciTikZ-Bench to provide strictly compilable training data and comprehensive evaluation across visual fidelity and code quality.SciTikZ-Bench contains diverse scientific figures and evaluates multiple structural patterns and domains.
  • Framework: Dual Self-Consistency reinforcement learning unifies visual fidelity and structural logic through round-trip reconstruction and closed-loop verification.The paradigm removes ground-truth dependence and supports logical self-consistency on unlabeled data.
  • Results: SciTikZer-8B achieves state-of-the-art performance, outperforming general-purpose and specialized baselines in compilation rates and visual alignment.The reported results include comparisons with substantially larger models and domain-specialized systems.

2 Related Work

Prior work advances visual program synthesis, TikZ generation, and verifiable reinforcement learning, but direct image-to-TikZ synthesis remains less explored. The paper situates its contribution at the intersection of these lines of research.

  • Visual Program Synthesis: Visual program synthesis has progressed from chart extraction toward complex reproduction and editing through imperative Python-library systems and dedicated benchmarks.Examples include ViperGPT, MatPlotAgent, METAL, DePlot, Plot2Code, ChartMimic, and ChartEdit.
  • Automated TikZ Generation: TikZ recovery differs from image vectorization because scientific diagrams require semantic topology rather than only unstructured geometric primitives.Direct image-to-TikZ synthesis remains underexplored because geometry must be inferred from pixels and data quality is limited.
  • Reinforcement Learning and Verifiable Generation: Verifiable-generation research uses reinforcement learning and render-and-compare feedback for code, mathematics, and visual reasoning tasks.Related approaches include GRPO, RRVF, VisionR1, Visual Sketchpad, GRIT, and OpenThinkIMG.

3 SciTikZ-230K Dataset

SciTikZ-230K is built through an execution-centric, coarse-to-fine purification pipeline that repairs, validates, and semantically filters heterogeneous graphics data. The resulting corpus contains 230K high-fidelity instances, while SciTikZ-Bench spans diverse domains and difficulty levels.

  • Data Engine: The MLLM-powered data engine combines Active Remediation with Coarse-to-Fine Purification to curate data from heterogeneous sources.The pipeline is designed to improve executability, alignment, and data utilization at scale.
  • Data Engine: MLLM-guided remediation corrects compilation faults and distills misaligned pairs while preserving diversity under strict visual-program alignment.This extends the pipeline beyond passive filtering of raw data.
  • Data Challenges: Raw sources contain non-executable code, missing dependencies, non-standard formatting, and visual-code misalignment that impede direct training.The dataset aggregates material from HuggingFace, TeX StackExchange, and arXiv.
  • Runtime Validation: The pipeline refactors non-standalone fragments, executes samples in a sandbox, and repairs syntax faults using compiler diagnostics.About 60% of faulty instances are recovered through the iterative remediation process.
  • Coarse-to-Fine Purification: Fine-grained adjudication scores correctness, layout, readability, scientific plausibility, and visual complexity before multi-criteria filtering.The aggregate quality score is defined as S_total = ∑s_i.
  • Outputs: SciTikZ-230K contains 230K high-fidelity, precisely aligned instances distributed across scientific domains, while SciTikZ-Bench covers Easy, Medium, and Hard schematic difficulties.The benchmark ranges from basic geometric primitives to complex schematics.

4 SciTikZer: A Faithful Img2TikZ Generator

SciTikZer combines supervised initialization, curriculum selection, and Dual Self-Consistency RL to generate executable TikZ with visual and structural fidelity. Its staged optimization grounds visual alignment before enforcing round-trip code consistency and filters self-consistency rewards by fidelity.

  • 4.1 Supervised Warm-up for Initialization: SciTikZer uses SFT on curated data, curriculum selection, and DSC RL to optimize visual fidelity and self-consistency.SFT initializes the policy, while curriculum selection focuses RL on challenging samples involving compilation errors or visual discrepancies.
  • 4.2 Curriculum Data Selection: 8K instances are selected for RL by excluding mastered samples with Svis > τmax and intractable outliers with Svis < τmin.The curriculum first filters by visual complexity and then targets samples within the model’s zone of proximal development.
  • 4.3.1 Stage 1: Visual Fidelity Alignment: Stage 1 uses GRPO with execution-gated, multi-granularity visual rewards combining semantic alignment and structural precision.Compilation failures receive a heavy penalty, while successful outputs are evaluated with SigLIP semantics and LPIPS-based geometric precision.
  • 4.3.1 Stage 1: Visual Fidelity Alignment: Hinge-scaled semantic similarity suppresses low-quality noise and amplifies gradients for high-fidelity samples above the baseline threshold τhold.The semantic score is based on cosine similarity between SigLIP embeddings of the source and rendered images.
  • 4.3.2 Stage 2: Self-Consistency Refinement: Stage 2 renders generated code, back-translates the image into reconstructed code, and rewards consistency using structural token similarity and CrystalBLEU.The round-trip mechanism uses the deterministic compiler to couple image-to-code and code-to-image directions within one policy.
  • 4.3.2 Stage 2: Self-Consistency Refinement: Self-consistency rewards activate only when visual alignment exceeds τgate, preventing degenerate code from receiving reinforcement through trivial render-and-reconstruct loops.The total reward adds the code-consistency term to the visual reward only under the fidelity gate.

5 Experiments

Experiments evaluate SciTikZer across curated benchmarks, external data, training stages, ablations, cross-language transfer, and human judgments. Results show strong compilation, visual fidelity, generalization, and structural consistency, with benefits from data curation and DSC.

  • Experimental Setup: SciTikZ-Bench evaluates zero-shot performance using visual-fidelity and code-quality metrics, while also reporting results on the established DaTikZ-v3 test set.The benchmark contains 611 manually verified, decontaminated samples and measures semantic alignment, structural precision, compilation, and code similarity.
  • Main Results: 97.2% compilation success makes SciTikZer-8B the SciTikZ-Bench leader, exceeding Gemini-2.5-Pro at 88.9% and Qwen3-VL-235B at 92.1%.It also surpasses DeTikZify-V2.5-8B in SigLIP semantic alignment, 93.8 versus 88.9, and LPIPS structural precision, 29.7 versus 37.3.
  • Main Results: On DaTikZ-v3, SciTikZer-8B achieves the highest compilation rate, 94.46%, and lowest KID, 1.14, while exceeding Qwen3-VL-235B in cBLEU and DSim.The reported comparisons are cBLEU 16.17 versus 16.05 and DSim 88.29 versus 83.91.
  • Progressive Training: Stage 1 Visual RL reduces 8B LPIPS from 51.4 to 37.0, while Stage 2 DSC-RL raises compilation to 97.2% and improves fine-grained perceptual metrics.The progression suggests render-based feedback improves geometric grounding, whereas DSC supplies further structural refinement despite a slight C-BLEU trade-off.
  • Ablation Study: For 8B models, curated SciTikZ-230K data increases compilation from 76.4% to 81.0% and raises SigLIP by 5.0 points compared with raw data.The curated dataset also performs better than DaTikZ-v3 on most metrics.
  • Ablation Study: DSC raises 8B compilation to 97.2% and lowers LPIPS from 30.8 to 29.7 relative to standard GRPO, prioritizing structural consistency over higher C-BLEU.In the 4B comparison, GRPO has C-BLEU 29.0 versus DSC’s 28.6, while DSC improves visual alignment.
  • Cross-Language Generalization: Applying DSC-RL to Python generation on ChartMimic outperforms standard RL in both executability and visual fidelity.The cross-language experiment uses VinciCoder-8B-SFT as the backbone, extending evaluation beyond declarative TikZ.
  • Human and Case Analysis: SciTikZer-8B receives the highest human preference, 59%, and aggregate human score, 12.08, with representative cases showing stronger structural and spatial fidelity.The case analysis highlights coordinate alignment, complex structural details, and fine-grained spatial relations.

6 Conclusion

The paper introduces a closed-loop TikZ synthesis framework combining curated data, multifaceted evaluation, and Dual Self-Consistency Reinforcement Learning. Its components target compilability, visual fidelity, and structural logic in scientific graphics generation.

  • The framework addresses graphics program synthesis for scientific figures by enabling multimodal models to generate TikZ code and using LaTeX render-and-compare feedback.
  • SciTikZ-230K provides a large-scale scientific graphics dataset, while SciTikZ-Bench evaluates synthesis across visual and structural dimensions.
  • Dual Self-Consistency Reinforcement Learning uses round-trip reconstruction and gated consistency checks to improve agreement between rendered images and generated code.
  • The appendix documents dataset construction, training implementation, evaluation details, additional analyses, and limitations and future work.

A.2 Data Preprocessing

The preprocessing pipeline converts heterogeneous TikZ snippets into self-contained, compilable, and visually faithful training examples. It combines normalization, diagnostic repair, heuristic filtering, automated fidelity adjudication, and taxonomy labeling.

  • The four-stage pipeline performs normalization and validation, diagnostic remediation, heuristic sanitization, and fidelity adjudication to enforce compilability and fidelity.The initial pool contains 310K snippets and is processed by this execution-centric pipeline.
  • Diagnostic Remediation: Diagnostic remediation supplies the reference image, failed code, and compilation logs to an MLLM that repairs syntax errors and missing dependencies with minimal edits.A representative repair adds the required backgrounds library and layer declarations without redesigning the drawing.
  • Diagnostic Remediation: Approximately 120K snippets fail initial runtime validation; remediation repairs about half, while the remaining 60K are discarded.
  • Heuristic Sanitization: Heuristic sanitization removes samples exceeding 8192 tokens or a 15:1 aspect ratio, redundant samples with more than five shared 50-grams, and external file dependencies.
  • Fidelity Adjudication and Taxonomy: Fidelity adjudication scores image-code pairs on correctness, layout, readability, scientific plausibility, and visual complexity, while taxonomy labeling organizes the dataset into 11 domains and 90+ subcategories.Both procedures use automated MLLM assessment with manual audits on 5% random subsets.

B Training Implementation Details

Training combines supervised fine-tuning with two sequential reinforcement-learning stages. The pipeline first improves executability and visual alignment, then adds gated self-consistency constraints to promote structurally coherent programs.

  • Supervised Fine-Tuning: SFT fine-tunes Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct to generate self-contained, compilable TikZ/LaTeX code from diagrams.
  • Reinforcement Learning: The RL implementation adds pdflatex-to-PNG rendering, SigLIP and LPIPS rewards, compilation signals, and gated self-consistency checks.
  • Two-stage RL: Stage 1 optimizes compilation success and visual alignment with render-level rewards, whereas Stage 2 adds consistency constraints only after visual fidelity exceeds a threshold.
  • Dual Self-Consistency RL: The dual self-consistency workflow samples candidate programs, renders them, and back-translates sufficiently faithful outputs into programs for code-level comparison.
  • Evaluation: Evaluation uses SciTikZ-Bench, a human-verified benchmark of 611 image-code pairs spanning 10 scientific categories, together with visual and structural metrics.

C.3 Evaluation Configuration

Evaluation is run with fixed inference settings and two aggregation modes that distinguish end-to-end reliability from quality conditional on successful compilation.

  • Inference Settings: Inference uses temperature 0.1, top-p 0.95, repetition penalty 1.05, and a 4,096-token maximum for all models under the same prompt template.
  • Metric Aggregation: The ALL mode assigns zero visual similarity or maximum perceptual-distance penalties to uncompilable or failed renders, capturing end-to-end reliability.
  • Metric Aggregation: The SUCCESS mode evaluates only successfully compiled outputs, measuring their upper-bound quality independently of compilation pass rate.

D.1 MLLM-Based Evaluation

The MLLM-based evaluation uses an automatic judge to score rendered predictions and their TikZ code for visual fidelity, structural correctness, and code quality. SciTikZer-8B achieves the strongest overall scores across all three criteria.

  • Evaluation Setup: Gemini-3-Flash evaluates predicted renderings and associated LaTeX/TikZ code using visual fidelity, structural correctness, and code quality criteria.Each criterion is scored as an integer from 0 to 10.
  • Results: 7.12 Visual Fidelity, 8.05 Structural Correctness, and 8.40 Code Quality are achieved by SciTikZer-8B, the strongest overall result in Table 12.The scoring guide ranges from 0 for unusable output to 9–10 for nearly perfect performance.
  • Evaluation Criteria: Visual fidelity measures image similarity in shapes, lines, relative sizes, and spatial arrangement.
  • Evaluation Criteria: Structural correctness measures preservation of topology, connectivity, hierarchy, directional flow, and relationships among major diagram components.
  • Evaluation Criteria: Code quality measures whether generated LaTeX/TikZ is clean, structured, semantically meaningful, interpretable, and free of hacky patterns.

D.2 Case Analysis

The case analysis examines model behavior across diverse scientific diagrams and a detailed irrigation-system schematic. SciTikZer-8B shows the strongest visual and structural performance, while baselines exhibit layout, compilation, or syntax problems.

  • Diverse Cases: 10 diverse scientific scenarios test generalization across geometric proofs, hierarchical flowcharts, and multi-layered optical diagrams.The cases contain varied structural and syntactical challenges.
  • Diverse Cases: Human evaluation finds SciTikZer-8B achieves the highest visual fidelity and structural accuracy, outperforming baselines with distorted layouts or compilation failures.
  • Detailed Schematic: In the irrigation-system schematic, baselines separately show reasonable outline reconstruction or compilable code, whereas Gemini-2.5-Pro fails from an undefined-anchor syntax hallucination.DeTikZify-v2.5-8B reconstructs the outline reasonably, while Qwen3-VL-Instruct-32B produces syntactically correct, compilable circuitikz code.

E.1 Limitations

The approach improves visual fidelity and execution success but remains constrained by training cost, environment sensitivity, and a functional-versus-lexical fidelity trade-off. The paper also identifies inference-time correction, sketch inputs, and broader graphics languages as future directions.

  • Limitations: The dual self-consistency RL loop has higher computational overhead than standard SFT because training requires multiple forward passes and external LaTeX rendering.
  • Limitations: TikZ generation remains sensitive to rendering environments, so macro-package and compiler differences can produce subtle cross-platform discrepancies.
  • Limitations: The optimization objective favors functional correctness over lexical similarity, allowing visually correct code to depart from ground-truth coding styles or idioms.
  • Future Directions: Future directions include inference-time self-correction, sketch-to-TikZ synthesis, and scaling to broader formal graphics languages.
Loading 2604.06079v1…