PaperlayerView exact revision on arXiv

Source-linked AI summary

AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ

Jonas Belouadi, Anne Lauscher, Steffen Eger

arXiv:2310.00367v2cs.CLcs.CV

TL;DR

Scientific figure generation needs precise, legible vector graphics, but direct generation from low-level primitives is difficult. AutomaTikZ uses TikZ as an intermediate representation, introduces the 120k-example DaTikZ dataset, and fine-tunes LLaMA and CLiMA with multimodal CLIP embeddings. Both fine-tuned models outperform GPT-4 and Claude 2 in similarity to human-created figures, while CLiMA additionally improves text-image alignment.

  • Problem

    Scientific figures require precise, legible vector graphics, whereas existing text-to-image systems mainly generate raster graphics and vector methods often struggle with geometric relations or complex outputs.

  • Method

    AutomaTikZ uses TikZ as an intermediate representation, releases the paired DaTikZ dataset, and fine-tunes LLaMA and CLiMA with multimodal CLIP embeddings.

  • Results

    Fine-tuned LLaMA and CLiMA outperform GPT-4 and Claude 2 in similarity to human-created figures, with CLiMA additionally improving text-image alignment.

  • Takeaways & Limitations

    AutomaTikZ provides a publicly available framework, model weights, and datasets for text-guided scientific vector-graphics generation.

  • Takeaways & Limitations

    The dataset pipeline excludes over 120k TikZ images that fail to compile, and arXiv licensing prevents redistributing some extracted drawings.

Abstract

from arXiv · show

Generating bitmap graphics from text has gained considerable attention, yet for scientific figures, vector graphics are often preferred. Given that vector graphics are typically encoded using low-level graphics primitives, generating them directly is difficult. To address this, we propose the use of TikZ, a well-known abstract graphics language that can be compiled to vector graphics, as an intermediate representation of scientific figures. TikZ offers human-oriented, high-level commands, thereby facilitating conditional language modeling with any large language model. To this end, we introduce DaTikZ, the first large-scale TikZ dataset consisting of 120k TikZ drawings aligned with captions. We fine-tune LLaMA on DaTikZ, as well as our new model CLiMA, which augments LLaMA with multimodal CLIP embeddings. In both human and automatic evaluation, CLiMA and LLaMA outperform commercial GPT-4 and Claude 2 in terms of similarity to human-created figures, with CLiMA additionally improving text-image alignment. Our detailed analysis shows that all models generalize well and are not susceptible to memorization. GPT-4 and Claude 2, however, tend to generate more simplistic figures compared to both humans and our models. We make our framework, AutomaTikZ, along with model weights and datasets, publicly available.

1 Introduction

Scientific figures require precise, legible vector graphics, but existing generation methods struggle with complex, geometrically accurate outputs. AutomaTikZ addresses this gap with TikZ, the DaTikZ dataset, and LLaMA-based models including CLiMA.

  • Motivation: Existing vector-generation methods often lose geometric relations or produce only limited-complexity outputs such as icons and font characters.
  • Motivation: AutomaTikZ uses TikZ as a high-level intermediate representation for generating complex scientific vector graphics from captions.TikZ abstracts lower-level vector formats through constructs that can be compiled into vector graphics.
  • Contributions: DaTikZ provides approximately 120k paired TikZ drawings and captions, enabling fine-tuning of language models for scientific figure generation.
  • Results: Fine-tuned LLaMA produces figures more similar to human-created figures than GPT-4 and Claude 2 in both automatic and human evaluations.
  • Contributions: CLiMA augments LLaMA with multimodal CLIP embeddings, improving text-image alignment and benefiting from supplementary image inputs.
  • Results: GPT-4 and Claude 2 generate simpler outputs than LLaMA and CLiMA, sometimes copying captions into images, while the models generally produce novel outputs.

2 Related Work

Related work spans text-to-image generation, vector graphics, scientific figure understanding, and code generation. Existing approaches generally leave scientific vector-figure generation insufficiently addressed.

  • Scope: The paper connects these four areas to position TikZ-based scientific figure synthesis within existing generation and understanding research.
  • Text-to-Image & Vector Graphics Generation: Text-to-image systems have progressed through GAN, autoregressive, and diffusion models, but remain limited to raster graphics for scientific figures.
  • Text-to-Image & Vector Graphics Generation: Vector-graphics methods generate characters, icons, or vectorized text-to-image outputs, but commonly lack text conditioning or struggle with flat-colored scientific figures.
  • Scientific Figure Understanding: Scientific figure understanding has focused substantially on captioning and visual question answering, the inverse direction of figure generation.
  • Code Generation: AutomaTikZ is also related to code generation because graphics languages are programming-language subsets, although code-generation research mainly targets high-resource programming languages.

3 The DaTikZ Dataset

DaTikZ is a large paired dataset of TikZ drawings and captions assembled from diverse online and artificial sources. Its construction emphasizes compilability, caption augmentation, and broad coverage.

  • Dataset Overview: DaTikZ is presented as the first large-scale dataset pairing TikZ drawings with corresponding captions.
  • Data Acquisition: The dataset combines curated websites, repositories, TEX Stack Exchange, arXiv papers, and artificial examples, retaining drawings that compile with TEX Live 2023.
  • Dataset Statistics: The dataset overview reports both total size and the percentage of augmented data for the full collection and each source.
  • Data Acquisition: ArXiv extraction expands includes, preserves required preamble elements and macros, and excludes approximately 120k drawings that fail compilation.
  • Data Augmentation: Short captions are automatically augmented with LLaVAR descriptions, selecting among five candidates using CLIPScore before concatenating the top-ranked description.

4 Methods

The methods fine-tune LLaMA to generate TikZ code from captions and introduce CLiMA by injecting CLIP visual representations into LLaMA. They also address compilation failures through iterative resampling.

  • LLaMA: LLaMA is fine-tuned with DaTikZ captions as inputs and TikZ code as ground-truth outputs.The original LLaMA release is used partly because later models have less clearly specified training data, complicating leakage control.
  • CLiMA: CLiMA combines LLaMA with CLIP to provide visual interpretations of captions and support rasterized scientific figures as additional inputs.
  • CLiMA: CLIP’s multimodal projection layer places text and image information in a common embedding space, enabling visual information to condition TikZ generation.
  • CLiMA: The system connects CLIP outputs to LLaMA inputs through soft prompting and a feed-forward projection layer between their embedding spaces.
  • Compilation Correction: Because TEX syntax is difficult to constrain with grammar-based decoding, the method iteratively resamples code around compilation errors using logfile diagnostics.

5 Experiments

The experiments evaluate fine-tuned LLaMA and CLiMA models against proprietary baselines using automatic metrics and human best-worst scaling. CLiMA models generally outperform their LLaMA counterparts and GPT-4 or Claude 2, while reference images improve some similarities but not overall image quality.

  • Automatic Evaluation: 13b models outperform 7b models by 0.2–0.5pp on all string-similarity and image-based metrics, while compilation sampling rates remain comparable.
  • Automatic Evaluation: CLiMAimg improves CLIPScoreimg by 1.2pp over CLiMA13b and raises string-based metrics by 0.1–0.4pp, without improving CLIPScore or KID.
  • Automatic Evaluation: GPT-4 and Claude 2 score up to 2.1pp higher on CLIPScore but have up to 3.9pp lower EED, up to 6.6pp lower CLIPScoreimg, and more than six times larger KID.
  • Automatic Evaluation: CLiMA7b and CLiMA13b outperform their respective LLaMA models in five of seven metrics, while GPT-4 and Claude 2 substantially underperform them.
  • Human Evaluation: Human evaluation ranks CLiMA13b above LLaMA13b for caption similarity and CLiMAimg above LLaMA13b for reference similarity, while GPT-4 trails on both.

6 Analysis

The analysis tests code novelty and caption copying to distinguish memorization from degenerate text-image alignment. Models generate increasingly novel code for longer n-grams, but GPT-4 and Claude 2 copy captions more frequently than the proposed models.

  • Novelty and Copying: For n>8, more than 80% of model code n-grams are novel, a pattern associated with models not memorizing their training data.
  • Novelty and Copying: For 1-grams, CLiMA13b and LLaMA13b copy around 6.5% of caption n-grams, compared with more than 10% for GPT-4 and Claude 2.
  • Novelty and Copying: For n>5, the proposed models and humans nearly stop copying captions, whereas Claude 2 and especially GPT-4 continue copying with an almost linear trend.
  • Novelty and Copying: The copying results reinforce CLIPScoreimg as a more robust metric for text-rich images because it is less susceptible to typographic attacks.

7 Conclusion & Future Work

AutomaTikZ combines a paired TikZ-caption dataset with CLiMA, which integrates multimodal CLIP embeddings into LLaMA for scientific vector-graphic generation. The authors report improved performance over LLaMA and proprietary baselines, while identifying typographic attacks and incomplete data extraction as important boundaries.

  • Conclusion: AutomaTikZ releases DaTikZ, a large paired TikZ-caption dataset, and CLiMA, a LLaMA architecture augmented with multimodal CLIP embeddings.
  • Conclusion: Fine-tuned CLiMA outperforms LLaMA on several metrics and surpasses proprietary GPT-4 and Claude 2, while also accepting images as inputs.
  • Conclusion: Integrating CLIP can improve visual-concept tasks even with text-only inputs, but evaluation of text-rich images must account for typographic attacks.
  • Future Work: The extraction pipeline excluded over 120k TikZ images that failed to compile, motivating future improvements and richer source-document inputs.

8 Ethics Statement

The dataset release is constrained by licensing and non-commercial-use restrictions, while comparisons with proprietary models limit reproducibility. The authors also caution that generated figures may inherit model and data limitations and could support misuse.

  • Dataset and licensing: DaTikZ excludes arXiv drawings under non-exclusive licenses from its public release, although the dataset-generation code can recreate the full version.Most collected sources use licenses permitting copying and redistribution.
  • Dataset and licensing: GPT-4-generated samples are restricted to non-commercial applications because OpenAI prohibits using GPT-4 to create competing services.
  • Evaluation limitations: Comparisons with GPT-4 and Claude 2 are hindered because their unpublished training details and hyperparameters reduce reproducibility.
  • Model risks: The models may inherit biases, flaws, or gaps from their base models and training data, misinterpret inputs, fabricate details, or overlook information.
  • Model risks: AutomaTikZ could be misused to generate disinformation and fake science despite being designed to assist legitimate scientific-figure production.

A Code & Image Complexity

The analysis links TikZ code length to generated-image complexity and compares automatic evaluation scores across models. It also describes prompt strategies, evaluation setup, and resource constraints affecting training and reproducibility.

  • A Code & Image Complexity: Humans score over twice as high as GPT-4 on image complexity, with CLiMA13b and LLaMA13b between them; CLiMA13b ranks above LLaMA13b.Image quality differences are less pronounced, while GPT-4 produces the simplest figures overall.
  • B Prompt Engineering: GPT-4 and Claude 2 use a chain-of-thought prompt that separates TikZ generation into a drawing guide, full code, and summary while discouraging caption copying.
  • C.1 Training Challenges: Training is limited to 1200-token contexts by GPU resources, preventing multi-turn compiler-log feedback and requiring more resource-efficient alternatives.
  • C.2 In-depth Evaluation Metric Scores: CLiMAimg and CLiMA13b generally outperform other models automatically, while GPT-4 and Claude 2 can score highly on CLIPScore through caption copying.The evaluation includes code, image, and caption-image metrics, plus compile-time error rate.
  • C.2 In-depth Evaluation Metric Scores: The experiments use additional ablations of caption augmentation, image forwarding to CLIP, and individual DaTikZ data sources to assess their contributions.

C.3 Ablation Studies

The ablations show that both caption augmentation and image sampling improve CLiMA performance, while data-source contributions vary substantially. ArXiv is most important overall, whereas curated examples provide unusually strong value relative to their size.

  • Impact of Data Augmentation: Removing caption augmentation lowers most metrics, especially CLIPScore by 1.1pp and CLIPScoreimg by 2.4pp, despite subtle CSR and KID improvements.The authors attribute this pattern to less caption information weakening caption-image alignment while allowing more flexibility on reference-free metrics.
  • Impact of Data Augmentation: Sampling images during training worsens most metrics, with only CSR remaining comparable, supporting image integration even for text-only evaluation.
  • Impact of Data Augmentation: Both augmentation techniques contribute positively: caption augmentation improves caption-image alignment, while image-based training improves overall performance.
  • Importance of each Data Source: Removing arXiv causes the largest cross-metric decline, including drops of 4.5pp in EED, 5.5pp in CLIPScoreimg, 1.4pp in CLIPScore, and 1.9pp in CrystalBLEU.ArXiv contributes 86k scientific-figure examples; CER nearly quadruples and CSR also increases when it is removed.
  • Importance of each Data Source: Curated examples have a particularly strong positive effect relative to their size, whereas TEX Stack Exchange produces comparatively small changes on the test set.The authors attribute curated examples’ disproportionate value to likely higher-quality code and captions.

D Annotator Demographics

The evaluation used expert annotators with strong English proficiency and science or technology research backgrounds. Crowdworker judgments showed very low agreement with experts and among crowdworkers, so crowdsourcing was discontinued.

  • Annotator Demographics: The expert panel comprised faculty members, PhD students, and assistants from other institutions, all with science or technology backgrounds and research experience.
  • Annotator Demographics: All expert annotators had English proficiency at C1 level or above under the Common European Framework of Reference for Languages.
  • Annotator Demographics: Crowdworker ratings correlated strikingly little with expert ratings, with ρ<0.1, and crowdworkers also showed low agreement among themselves.
  • Annotator Demographics: Because crowd judgments were unreliable relative to expert judgments and internally, the authors discontinued further crowdsourcing experiments.

E Examples

The examples illustrate human-created DaTikZ drawings and model-generated scientific figures spanning high- and low-rated outputs. They also expose practical generation issues, including caption copying, nested TikZ environments, and excessive verbosity.

  • Examples: Figure 6 contrasts high- and low-rated outputs from CLiMA13b, LLaMA13b, and GPT-4 according to expert judgments of caption similarity.Augmented caption sections are emphasized in the examples.
  • Examples: GPT-4 examples include caption copying, where text from the input caption is visibly reproduced in the generated figure.Additional caption-copying cases are shown in Figure 7, alongside CLiMA13b outputs for the same captions.
  • Examples: Figure 5 presents human-created DaTikZ drawings drawn from dedicated TikZ websites, repositories, and arXiv.
  • Examples: For an overlay example, both GPT-4 and CLiMA13b correctly use the overlay option, but GPT-4 nests tikzpicture environments and CLiMA13b produces notably verbose code.
Loading 2310.00367v2…