PaperlayerView exact revision on arXiv

Source-linked AI summary

TikZilla: Scaling Text-to-TikZ with High-Quality Data and Reinforcement Learning

Christian Greisinger, Steffen Eger

arXiv:2603.03072v3cs.AIcs.CLcs.CV

TL;DR

Text-to-TikZ research is constrained by small, noisy datasets and captions that do not reliably specify rendered figure semantics. The paper introduces DaTikZ-V4 and trains small TikZilla Qwen models with SFT followed by RL using an inverse-graphics image encoder for rewards. TikZilla outperforms GPT-4o across automatic and human evaluation and matches GPT-5 in image-based evaluation, while using much smaller models.

  • Problem

    Existing Text-to-TikZ datasets are too small and noisy, while captions and SFT-only training provide limited supervision for rendered semantics.

  • Method

    DaTikZ-V4 combines large-scale TikZ data, LLM debugging, and VLM-generated descriptions; TikZilla applies SFT followed by RL with an inverse-graphics image encoder as reward model.

  • Results

    TikZilla outperforms GPT-4o across automatic and human evaluation and matches GPT-5 in image-based evaluation at much smaller model sizes.

  • Takeaways & Limitations

    Small open-source models can achieve strong Text-to-TikZ performance when trained with higher-quality data and domain-specific visual rewards.

  • Takeaways & Limitations

    Automatically generated VLM descriptions may omit or hallucinate details, and reward optimization can reinforce errors when descriptions diverge from figures.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to assist scientists across diverse workflows. A key challenge is generating high-quality figures from textual descriptions, often represented as TikZ programs that can be rendered as scientific images. Prior research has proposed a variety of datasets and modeling approaches for this task. However, existing datasets for Text-to-TikZ are too small and noisy to capture the complexity of TikZ, causing mismatches between text and rendered figures. Moreover, prior approaches rely solely on supervised fine-tuning (SFT), which does not expose the model to the rendered semantics of the figure, often resulting in errors such as looping, irrelevant content, and incorrect spatial relations. To address these issues, we construct DaTikZ-V4, a dataset more than four times larger and substantially higher in quality than DaTikZ-V3, enriched with LLM-generated figure descriptions. Using this dataset, we train TikZilla, a family of small open-source Qwen models (3B and 8B) with a two-stage pipeline of SFT followed by reinforcement learning (RL). For RL, we leverage an image encoder trained via inverse graphics to provide semantically faithful reward signals. Extensive human evaluations with over 1,000 judgments show that TikZilla improves by 1.5-2 points over its base models on a 5-point scale, surpasses GPT-4o by 0.5 points, and matches GPT-5 in the image-based evaluation, while operating at much smaller model sizes.

1 INTRODUCTION

Text-to-TikZ systems are limited by small, noisy datasets, weak captions, and the absence of direct visual feedback. The paper addresses these issues with DaTikZ-V4, richer descriptions and debugging, an inverse-graphics reward model, and small TikZilla models trained with SFT and RL.

  • Motivation: TikZ is precise and interpretable but difficult for humans and LLMs because of its steep learning curve and varied syntax.Prior caption–TikZ approaches remain constrained by sparse data, noisy captions, non-executable code, and limited visual feedback.
  • Contributions: DaTikZ-V4 contains over 2M unique TikZ samples, quadrupling the scale of prior datasets.The dataset is sourced from newer arXiv submissions and GitHub.
  • Contributions: The dataset improves quality through dynamic package filtering, VLM-generated figure descriptions, and LLM debugging of uncompilable TikZ code.These components target noisy captions, invalid programs, and missing visual supervision.
  • Contributions: An image encoder trained on Image-TikZ provides semantically meaningful rewards for reinforcement learning.The reward model is intended to expose optimization to rendered figure semantics.
  • Contributions: TikZilla uses small open-source Qwen models trained with SFT followed by RL, outperforming GPT-4o across automatic and human evaluation and matching GPT-5 in image-based evaluation.The released models are 3B and 8B variants.

2 RELATED WORK

Prior scientific graphics systems evolved from heuristic and neural approaches toward LLM-based caption-to-program generation, while reinforcement learning increasingly aligns models with preferences, execution feedback, and multimodal objectives.

  • Text-Guided Graphics Program Generation: LLM-based graphics methods include AutomaTikZ for caption–TikZ pairs and StarVector for scientific vector-graphics generation.Earlier heuristic and neural sequence methods struggled with complex scientific figures.
  • Post-training with Reinforcement Learning: Recent reinforcement-learning methods align language models with human preferences, verifiable tasks, execution feedback, mathematical reasoning, and multimodal objectives.The cited examples include GRPO, RLEF, Yoshihara et al., and VisionR1.

3 CAPTION QUALITY ANALYSIS

Existing captions often omit structural details needed for figure reconstruction, whereas VLM-generated descriptions are richer and more faithful to the figures.

  • Caption Quality Analysis: Most DaTikZ-V3 captions omitted figure types, components, or labels and received low usefulness scores of 1–2 from annotators.The analysis covered 200 samples and used three annotators.
  • Caption Quality Analysis: VLM-generated descriptions were evaluated against human-written references using BLEU-4, ROUGE-L, and semantic textual similarity.The comparison included multiple VLMs, raw captions, and human-written descriptions.
  • Caption Quality Analysis: GPT-4o descriptions reached 0.089 BLEU-4 versus 0.003 for raw captions, approaching 0.094 human-human agreement.VLM outputs were also 120–170 characters long versus 34 characters for raw captions.

4 DATASET

DaTikZ-V4 expands Text-to-TikZ data scale and quality through broader sourcing, filtering, debugging, and semantically rich VLM descriptions.

  • DaTikZ-V4 addresses the need for larger, higher-quality Text-to-TikZ data to support stronger models.
  • Data Sourcing: Over 400,000 unique TikZ samples come from approximately 5,500 GitHub repositories, making GitHub nearly as large as all of DaTikZ-V3.
  • Filtering: The dataset broadens extraction beyond tikzpicture to tikz-cd and circuitikz, recursively splits subfigures, standardizes code, and detects packages dynamically.
  • LLM Debugging: LLM debugging repairs 600K of 1.3M uncompilable samples in the first pass, substantially increasing usable TikZ programs.
  • VLM-based Image Description: VLMs annotate around 1.3M compilable samples with precise descriptions because raw captions can be unhelpful for figure reproduction.

5 METHOD

TikZilla combines SFT for TikZ syntax with GRPO-based RL that uses image-derived semantic rewards, while constraining format and mitigating reward hacking.

  • Training Pipeline: TikZilla uses two-stage training: SFT grounds syntax and token distributions, then RL incorporates rendered-image feedback for visual alignment.
  • Stage 1: Supervised Finetuning: SFT ensures syntactic validity and prompt alignment but remains unaware of rendered semantics, causing loops, irrelevant content, and incorrect spatial relations.
  • Stage 2: Reinforcement Learning: GRPO samples multiple outputs per description, scores them with scalar rewards, and updates the policy using group-centered advantages.
  • Rewards: The image encoder remains frozen during RL to mitigate reward hacking, addressing concerns that CLIPScore and DreamSim poorly capture scientific-figure nuances.
  • Rewards: The reward model uses DeTikZify-V2’s image encoder and Earth Mover’s Distance over patch embeddings to measure semantic similarity between ground-truth and predicted images.
  • Rewards: The semantic reward lies in [0, 1], while a format reward assigns zero to outputs lacking valid TikZ document environments.

6 EXPERIMENTS

Experiments evaluate TikZilla and competing models on a contamination-free 1,047-sample test set using alignment, fidelity, code, executability, efficiency, and independent perceptual metrics.

  • Experimental Setup: The test set contains 1,047 contamination-free samples filtered by date, source uniqueness, n-gram matching, and manual inspection.
  • Models: Nine LLMs are benchmarked, including GPT-5, GPT-4o, open-source baselines, TikZero-Plus-10B, and TikZilla SFT and RL variants.
  • Evaluation Metrics: Evaluation covers text–image alignment, perceptual fidelity, code similarity, compilation, and token efficiency through CLIP, DreamSIM, TED, CR, and AT.
  • Evaluation Metrics: DINOScore and LPIPS provide metrics independent of the domain-specific reward model for RL ablations.

7 RESULTS

TikZilla’s SFT+RL models achieve the strongest reported automatic performance and competitive human ratings, while ablations show benefits from high-quality descriptions, debugging, dataset scale, and domain-specific rewards.

  • Main Results: 0.385 AVG is achieved by TikZilla-3B-RL and 0.384 by TikZilla-8B-RL, both surpassing GPT-5 at 0.365 on automatic evaluation.TikZilla-3B-RL also improves over TikZero-Plus-10B by +0.085 CLIP, +0.334 DSim, 37% compilation rate, and 261 fewer tokens.
  • Human Evaluation: GPT-5 scores highest on textual alignment at 4.18, while TikZilla-8B-RL ties it on image alignment at 3.48 versus 3.46.TikZilla-3B-RL reaches 3.40 text and 3.30 image, and RL boosts the two Qwen models by +0.75 and +0.67 points.
  • Captions vs. Descriptions: VLM-generated descriptions outperform captions, with GPT-4o reaching 0.315 AVG versus 0.270, while mixing or oversampling captions degrades performance.For Qwen2.5-3B, descriptions yield 0.289 AVG versus 0.279 with captions.
  • LLM-Based Debugging: Training only on first-try compilable code reduces performance from 0.324 to 0.288, demonstrating the value of LLM-based debugging.RL also raises compilation rates to 95–98% and reduces token length without an explicit efficiency reward.
  • Dataset Sizes and Model Size: Performance rises sharply through the first 25% of DaTikZ-V4 and then improves more gradually toward the full dataset.The smaller Qwen2.5-3B can slightly outperform Qwen3-8B after SFT+RL, whereas its low 0.202 baseline shows stronger dependence on SFT syntax grounding.

8 CONCLUSION, LIMITATIONS, AND FUTURE WORK

The paper presents DaTikZ-V4 and TikZilla as a high-quality data and SFT+RL framework for efficient Text-to-TikZ generation. It identifies automatically generated descriptions as a key limitation and calls for stronger annotation, rewards, metrics, and broader structured-generation applications.

  • Conclusion: DaTikZ-V4 combines large-scale data, LLM debugging, VLM descriptions, and an inverse-graphics reward model in a two-stage SFT+RL framework.TikZilla models achieve near-perfect compilation rates and surpass GPT-4o across automatic and human evaluation.
  • Conclusion: TikZilla demonstrates that small open models can support reproducible and efficient text-to-image generation while reducing reliance on costly proprietary systems.
  • Limitations and Future Work: Automatically generated VLM descriptions may omit or hallucinate details, allowing errors to bias training and, rarely, reward optimization.The paper identifies more reliable annotation, fine-grained rewards, better human-aligned metrics, and extension to other structured tasks as future directions.

A.1 CAPTION QUALITY ANALYSIS

The caption-quality analysis used human annotation and agreement checks to assess whether existing captions contain the structural detail needed for figure reconstruction. It found that caption quality is limited and motivated richer descriptions.

  • Evaluation Setup: Three annotators evaluated 200 DaTikZ-V3 figures, with one annotator producing reference descriptions for BLEU-4, ROUGE-L, and STS scoring.The sample contained 74% arXiv and 26% TeX SE figures.
  • Agreement: Inter-annotator agreement was unweighted κ = 0.35 and weighted κ = 0.63 across the additional descriptions.The structural taxonomy was adapted from academic-writing practices and prior research taxonomies.

A.2 DATASET

DaTikZ-V4 augments TikZ data with synthetic examples, LLM-based debugging, and VLM-generated descriptions designed for precise visual reconstruction. The pipeline uses iterative repair and structured prompting to improve usable training data.

  • Synthetic Data: Synthetic data uses 2,000 templates, each producing 10 TikZ-generating queries through GPT-4o with minimal human intervention.
  • LLM Debugging: Qwen3-32B recovers 49.40% of debugging errors in one pass and 59.04% after three repair rounds, outperforming smaller tested variants.The evaluation covered 753 samples from all data sources.
  • VLM-Based Image Description: Qwen2.5-VL-7B-Instruct generates descriptions for all DaTikZ-V4 figures using few-shot exemplars and two days of processing on four A100 GPUs.
  • VLM-Based Image Description: The description prompt requires continuous prose specifying geometry, labels, colors, spatial relations, coordinates, dimensions, and orientation for precise TikZ reconstruction.

A.3 METHOD

TikZilla combines large-scale data, syntax-oriented supervised fine-tuning, and image-informed reinforcement learning to improve scientific figure generation. The evaluation compares model quality, compilation, code structure, and category-specific performance across human and automatic assessments.

  • A.3 METHOD: 1.3M Image–TikZ pairs from DaTikZ-V4 are used to finetune DeTikZify-V2 with 448×448 images and 2048-token outputs.Training runs for two epochs with AdamW, cosine scheduling, 3% warmup, batch size 128, and four H200 GPUs for 12 days.
  • A.5 RESULTS: 0.320 AVG is achieved using arXiv plus GitHub data, improving over 0.305 from arXiv alone, while TeX SE and synthetic data provide marginal benefits.The ablation identifies naturally occurring arXiv and GitHub TikZ as the most valuable data sources.
  • A.5 RESULTS: TikZilla almost always produces syntactically valid code, whereas GPT-5 frequently fails to compile complex figures because of missing imports, incorrect macro nesting, or hallucinated commands.TikZilla generally uses interpretable primitives such as \node, \draw, and \fill; GPT-5 often uses more elaborate, brittle constructs.
  • A.5 RESULTS: TikZilla performs best on geometrically constrained figures, while GPT-5 performs better on conceptual and loosely specified network-style diagrams.The reported categories include charts, function plots, schematics, commutative diagrams, and network-style figures.
  • A.5 RESULTS: RL substantially improves geometric coherence and spatial consistency, reducing misplaced labels and arrows relative to TikZilla without RL.The comparison concerns structural strengths that remain similar without RL but have more frequent spatial misalignments.
Loading 2603.03072v3…