PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal

arXiv:2310.12128v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Text-to-image models have been little explored for diagram generation, where dense spatial relationships and readable labels pose challenges. DiagrammerGPT uses LLM-generated plans to guide diagram generation and produces more accurate diagrams than baseline models.

  • Problem

    Text-to-image diagram generation remains underexplored, and existing models often lack fine-grained layout control for dense relationships and comprehensible text labels.

    Source: §1 · pp. 1–2

  • Method

    DiagrammerGPT uses an LLM planner-auditor loop to create and refine diagram plans, then generates diagrams from them with layout-guided DiagramGLIGEN and explicitly rendered labels.

    Source: §1 · p. 2

  • Results

    DiagrammerGPT generated more accurate diagrams than baselines in quantitative and qualitative analyses, and outperformed Stable Diffusion v1.4 in human preferences for image-text alignment and object relationships.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The framework may benefit educational applications such as presentation and paper creation, as well as human-in-the-loop diagram generation and modification.

    Source: G Limitations · p. 25

    DiagramGLIGEN sometimes fails to follow all aspects of otherwise generally correct diagram plans.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. A diagram is a symbolic/schematic representation that explains information using structurally rich and spatially complex visualizations (e.g., a dense combination of related objects, text labels, directional arrows/lines, etc.). Existing state-of-the-art T2I models often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows/lines, and also often fail to render comprehensible text labels. To address this gap, we present DiagrammerGPT, a novel two-stage text-to-diagram generation framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams. In the first stage, we use LLMs to generate and iteratively refine 'diagram plans' (in a planner-auditor feedback loop). In the second stage, we use a diagram generator, DiagramGLIGEN, and a text label rendering module to generate diagrams (with clear text labels) following the diagram plans. To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset. We show that our DiagrammerGPT framework produces more accurate diagrams, outperforming existing T2I models. We also provide comprehensive analysis, including open-domain diagram generation, multi-platform vector graphic diagram generation, human-in-the-loop editing, and multimodal planner/auditor LLMs.

1 Introduction

DiagrammerGPT addresses T2I models’ difficulty with densely related diagram objects and readable labels through LLM-generated plans and layout-guided generation. The paper also introduces AI2D-Caption and evaluates the framework’s accuracy and broader uses.

Motivation

  • Existing T2I models often produce incorrect object–arrow relationships and incomprehensible labels, including DALL-E 3, making information correctness more important than photorealism.The paper identifies insufficient fine-grained layout control as a central challenge for diagrams with densely connected objects.Source: §1 · pp. 1–2

Diagram planning

  • LLM planner–auditor feedback creates plans of entities, relationships, and 2D layouts, then refines errors such as incorrect positions or relationships.The auditor identifies potential errors and the planner updates the plan; the authors report that refinement can correct some small errors.Source: §1 · p. 2, Figure 2 · p. 3

Diagram generation

  • DiagramGLIGEN generates diagrams from plans using layout grounding for objects, text labels, and arrows, after which labels are explicitly rendered for readability.Built on GLIGEN and Stable Diffusion v1.4, DiagramGLIGEN is trained on AI2D-Caption and specialized for diagrams.Source: §1 · p. 2, Figure 1 · p. 2

Dataset

  • AI2D-Caption, built on AI2D, adds captions and object descriptions to address the lack of diagram datasets with detailed captions and fine-grained layout annotations.LLaVA 1.5 was used to create the annotations.Source: §1 · p. 3

Results

  • DiagrammerGPT generates more accurate diagrams than baseline models and outperforms Stable Diffusion v1.4 in human preference for image-text alignment and object relationships.The authors report that GPT-4-generated plans are quite accurate before and after refinement, while DiagramGLIGEN sometimes makes generation errors.Source: §1 · p. 3

Additional analyses

  • Additional analyses cover unseen domains, rendering plans in PowerPoint, Inkscape, and Adobe Illustrator, human-edited plans, and GPT-4Vision planners and auditors.The authors report that plans and diagrams can often be semantically accurate for unseen domains such as geology and plants.Source: §1 · p. 3

2 Related Works

Prior T2I models struggle with diagrams because they lack precise control over dense layouts and often render illegible labels. LLM-guided image-layout methods and TikZ-based generation also have limitations for diagram-specific control and diverse entities.

2.1 Text-to-Image Generation

  • T2I models struggle with diagrams because of distribution gaps, imprecise control of dense layouts, and frequently illegible text labels.Recent multimodal language and diffusion models have demonstrated impressive zero-shot photorealism.Source: §2.1 · pp. 3–4

2.2 Text-to-Image Generation with LLM-Guided Layouts

  • LLM-guided image-layout methods target natural images and lack diagram-specific text and dense-layout control; AutomaTikZ (Belouadi et al., 2023) is limited by TikZ primitives.In contrast, DiagrammerGPT supports diverse entities, including animals in life-cycle diagrams and planets or stars in astronomy diagrams.Source: §2.2 · p. 4

3 DiagrammerGPT: Method Details

DiagrammerGPT uses an LLM planning stage followed by layout-guided diagram generation and explicit text-label rendering. A planner–auditor feedback loop refines the plan before the generator renders it.

3.1 Stage 1: Diagram Planning

  • GPT-4 creates plans from 10 in-context examples, specifying entities, their relationships, and 2D bounding boxes.These plans provide the layout guidance for downstream diagram generation.Source: §3 · p. 4, §3.1 · p. 5
  • An auditor checks plans for mismatches with the prompt and gives feedback for iterative planner revisions, for up to four iterations in the experiments.The auditor uses GPT-4 with a different preamble and in-context examples than the planner.Source: §3.1 · p. 5

3.2 Stage 2: Diagram Generation

  • After DiagramGLIGEN generates an image from the plan, Pillow renders the plan’s text labels onto the diagram for clear, readable labels.The method renders labels explicitly because existing text-to-image models struggle to produce high-quality text.Source: §3 · p. 4, §3.2 · p. 5
  • DiagramGLIGEN adds gated self-attention layers to Stable Diffusion v1.4 and uses layout inputs for objects, labels, and relationships to guide diagram generation.During training, label and relationship inputs help reduce unreadable text and redundant arrows or lines at inference, when text labels are omitted.Source: §3.2 · p. 5

4 Experimental Setup

The experimental setup introduces AI2D-Caption, compares DiagrammerGPT with text-to-diagram baselines, and specifies automatic and human evaluation protocols.

4.1 AI2D-Caption Dataset

  • AI2D-Caption augments AI2D diagrams with LLaVA 1.5-generated diagram captions and bounding-box region descriptions.The original annotations were often brief or labeled objects only as “blob.”Source: §4.1 · p. 5, Figure 4 · p. 6
  • The manually annotated evaluation set contains 30 in-context examples and 75 test diagrams across astronomy, biology, and engineering.All 105 diagrams were held out from training; the in-context and test splits contain 10 and 25 diagrams per domain, respectively.Source: §4.1 · p. 6

4.2 Baseline Models

  • Baselines are Stable Diffusion v1.4, VPGen, and AutomaTikZ, evaluated zero-shot and fine-tuned on AI2D-Caption where applicable.For VPGen, both Vicuna13B and GLIGEN are fine-tuned.Source: §4.2 · p. 6

4.3 Evaluation Metrics

  • VPEval assesses object presence, counts, spatial and connection relationships, and text labels, using LLaVA 1.5 instead of BLIP-2.The authors report that LLaVA 1.5 was more faithful in diagram question answering in initial experiments and stronger overall than BLIP-2.Source: §4.3 · p. 6
  • Captioning evaluation compares LLaVA 1.5-generated diagram captions with ground truth using CIDEr and BERTScore; CLIPScore measures image-text and image-image similarity.The image-text comparison uses the original caption, the image-image comparison uses the ground-truth diagram, and both use OpenAI CLIP ViT-L/14.Source: §4.3 · pp. 6–7

4.4 Human Evaluation

  • An expert rates object presence and relationships in both pipeline stages on a 1–5 Likert scale for 25 layouts or images.Source: §4.4 · p. 7
  • Crowd-sourced annotators compare DiagrammerGPT with AI2D-Caption-fine-tuned Stable Diffusion v1.4 on 50 diagrams for image-text alignment and object relationships.Twenty unique Amazon Mechanical Turk annotators participate; the baseline was selected as the closest and strongest among the baselines.Source: §4.4 · p. 7

5 Results and Discussion

DiagrammerGPT outperforms the reported baselines on quantitative diagram-generation measures. Further analyses examine its error modes, qualitative outputs, extension to unseen domains and editable vector platforms, and planner LLM choices.

5.1 Quantitative Results

  • DiagrammerGPT outperforms zero-shot and fine-tuned baselines on overall and skill-specific VPEval, captioning, and CLIPScore.For CIDEr, it scores 26.4 vs. 4.2 for fine-tuned VPGen and 26.4 vs. 18.2 for fine-tuned Stable Diffusion v1.4; its two CLIPScores are 32.1 vs. 30.1 and 73.9 vs. 68.1 against fine-tuned Stable Diffusion v1.4, and it also achieves a few higher BERTScore points.Source: §5.1 · pp. 7–8

5.2 Human Evaluation

  • Plans scored 4.96 for object presence and 4.56 for object relations before refinement; refinement raised relations to 4.72, but generation reduced both scores.The authors suggest a stronger layout-guided image-generation backbone could improve final diagram accuracy.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Qualitative Analysis

  • In the earth-phases example, DiagrammerGPT follows the prompt, while Stable Diffusion v1.4 over-generates objects and AutomaTikZ fails to produce a proper diagram.Plans are generally correct, but DiagramGLIGEN sometimes fails to follow every aspect; the authors note that a better backbone could help.Source: §5.3 · pp. 8–9, Figure 5 · p. 8
  • Plan refinement can fix minor errors, including missing circuit connections and objects placed in the wrong locations.Examples are provided in the appendix.Source: §5.3 · p. 9

5.4 Additional Analysis: Open-Domain, Open-Platform, and Human-in-the-Loop

  • For an unseen earth-science domain, the planner produces fairly accurate layouts that DiagramGLIGEN can render into diagrams.Compared with DiagrammerGPT, DALL-E 3 tends to add redundant objects or text, can misrepresent prompt details such as Earth layers, and is noted to produce inaccurate diagram information.Source: §5.4 · p. 9, Figure 6 · p. 9
  • Diagram plans can drive editable vector diagrams in PowerPoint, Inkscape, and Adobe Illustrator, conveying key prompt information and layouts.The examples show inconsistent icon styles and limited icon retrieval, which users can address through editing.Source: §5.4 · p. 9, Figure 7 · p. 9
  • Human-in-the-loop editing lets users review generated layouts and move, add, remove, or resize objects before generating a revised diagram.The revised plan can remain in vector format or be returned to DiagramGLIGEN for a pixel-level result better suited to user requirements.Source: §5.4 · pp. 9–10, Figure 8 · p. 10

5.5 Different LLMs for diagram plan generation

  • GPT-4 and GPT-3.5 Turbo achieve overall planner scores of 78.4 and 74.7, respectively; GPT-4 is slightly better on object and text metrics and is used as the main LLM.LLaMA3 8B and fine-tuned Vicuna 13B each score below 30 overall.Source: Table 4 · p. 10, §5.5 · p. 10

6 Conclusion · Appendix · A LLM Prompt Templates and Diagram Plans

DiagrammerGPT uses LLM planning and refinement to generate diagrams with more semantically accurate layouts than baseline models. Its appendix describes prompt templates and plans that represent entities, relationships, and spatial layouts.

6 Conclusion

  • DiagrammerGPT’s two-stage framework uses LLMs to plan and refine diagrams, achieving more semantically accurate layouts than baseline models in quantitative and qualitative analyses.The paper also reports human error analysis, ablations, and analyses of open-domain and vector-graphic generation, human-in-the-loop editing, and multimodal planner/auditor LLMs.Source: §6 · p. 10

Appendix

  • The appendix covers prompt templates and plans, dataset collection, DiagramGLIGEN, experimental and human-evaluation setups, additional results and analyses, and limitations.Source: Appendix · p. 14

A LLM Prompt Templates and Diagram Plans

B AI2D-Caption Collection Details · C Additional DiagramGLIGEN Details · D Experimental Setups

AI2D-Caption uses LLaVA 1.5 to add captions and object-region descriptions to AI2D diagrams, with human evaluation finding the annotations adequate for DiagramGLIGEN domain adaptation. DiagramGLIGEN is trained on this dataset, and its gated self-attention incorporates grounding tokens for layout guidance.

B AI2D-Caption Collection Details

  • AI2D-Caption uses LLaVA 1.5 to add diagram captions and object-region descriptions to AI2D, whose original annotations target question answering and lack informative titles and per-object descriptions.LLaVA is prompted to describe the diagram and, separately, to describe each object in an image overlaid with labeled bounding boxes.Source: B AI2D-Caption Collection Details · pp. 14, 16
  • In a human evaluation of 50 annotations, LLaVA produced very good captions 80% of the time and very good bounding-box descriptions 68% of the time.Automatic annotation errors were generally minor for captions, while nearby boxes could receive identical descriptions; the test set and planner in-context annotations were manually annotated for correctness.Source: B AI2D-Caption Collection Details · p. 16

C Additional DiagramGLIGEN Details

  • DiagramGLIGEN starts from the GLIGEN Box+Text checkpoint and is trained on AI2D-Caption for 15k steps with batch size 5 per GPU.Training took 12 hours on 8 A6000 GPUs; the listed settings include AdamW, learning rate 5e-5, 2500 warmup steps, and 512x512 images.Source: C Additional DiagramGLIGEN Details · pp. 16–17
  • DiagramGLIGEN’s gated self-attention uses grounding tokens for layout-guided generation, whereas backbone cross-attention incorporates text tokens for text-guided generation [Li et al., 2023b].Source: C Additional DiagramGLIGEN Details · p. 17

D.1 Metrics

The metrics assess object presence, counts, spatial and connection relationships, and text-label correctness. Evaluation uses diagram plans to guide checks, with human evaluation also reported for relationships because the model can produce false positives.

D.1 Metrics

  • VPEval assesses object presence, exact object counts, spatial and connection relationships, and correct text labels through evaluation programs and modules (Cho et al., 2023b).Ground-truth diagram plans determine programs for object, count, and text evaluation; GPT-4 generates VQA questions for relationship checks.Source: §D.1 · p. 17
  • LLaVA 1.5 replaces BLIP-2 as the visual reasoner and is fine-tuned on relationships after initial tests revealed relationship-evaluation difficulties.The authors report that LLaVA 1.5 was more faithful in diagram question answering and stronger overall than BLIP-2 in initial experiments.Source: §D.1 · p. 17
  • Checks ask LLaVA whether each object is present, whether exactly N instances appear, and whether a spatial or connection relation holds; EasyOCR detects target text.Because LLaVA can produce false positives in relationship evaluation, the authors also report human evaluation of object relationships in main paper Sec. 4.4.Source: §D.1 · p. 17

E Human Evaluation Setup Details · F Additional Experiment Results

The human preference study compared DiagrammerGPT with fine-tuned Stable Diffusion v1.4 using five annotators per prompt, who judged object relationships and prompt alignment. Workers were screened for experience, approval rating, and English-language eligibility.

E Human Evaluation Setup Details

  • Five annotators compared shuffled DiagrammerGPT and fine-tuned Stable Diffusion v1.4 diagrams per prompt, judging object relationships and prompt alignment; their agreement was recorded.Source: E Human Evaluation Setup Details · pp. 17–18
  • Workers needed AMT Masters status, over 1,000 completed HITs, approval above 95%, and residence in the United States, Great Britain, Australia, or Canada.They received $0.06 per diagram pair, roughly $14–15 per hour.Source: E Human Evaluation Setup Details · p. 17
  • Object Relationships assessed object spacing, positioning, and arrows or lines, while Alignment assessed how well a diagram represented the input sentence.Source: E Human Evaluation Setup Details · p. 18

F.1 Qualitative Results and Analysis

Qualitative examples show DiagrammerGPT following prompts more closely than comparison systems, while auditor refinement improves plan layouts and connections. The final generator can still fail to realize parts of a plan.

Qualitative Results

  • DiagramGLIGEN sometimes fails to follow the plan, misdrawing the egg and swapping the larva and pupa; a better backbone could improve plan-based generation.These errors occur despite generally correct diagram plans.Source: §F.1 · pp. 18, 20
  • On AI2D-Caption examples, DiagrammerGPT plans strongly reflect prompts and final diagrams align more closely; open-domain examples show prompt adherence, while DALL-E 3 has superior visual quality but crowds objects.In the first AI2D-Caption example, DiagrammerGPT gets the object count right and has clear text.Source: §F.1 · p. 18, Figure 13 · p. 19, Figure 15 · p. 20

Diagram Plan Refinement

  • Auditor refinement corrects plan connections and layout, connecting the circuit switch to the battery and removing an overlap between the New Moon and Earth.The revised circuit has a more straightforward flow.Source: Figure 14 · p. 19, §F.1 · p. 20

F.2 Additional Analysis · F.3 Ablation Studies

The additional analyses examine DiagrammerGPT’s generalization, plan diversity, rendering options, and platform portability, while ablations describe how layout control and fine-tuning choices affect diagram generation. The results also compare GPT-4 with GPT-4Vision and GPT-4 with ground-truth plans.

F.2 Additional Analysis

  • DiagrammerGPT’s planner generalizes to unseen domains, while DALL-E 3 often produces crowded or prompt-inaccurate diagrams.Tests include geology and botany; examples also show plans for neural networks and vacation planning. DiagramGLIGEN adheres strongly to plans in some comparisons but struggles in others; the paper notes that the DALL-E 3 system card reports scientific inaccuracies in diagrams.Source: §F.2 · pp. 20–21, Figure 16 · p. 21
  • Diagram plans can be rendered as vector graphics in PowerPoint, Inkscape, and Illustrator using platform-specific scripts and retrieved icons.The implementation uses VBA for PowerPoint, a Python scripting extension for Inkscape, and JavaScript for Illustrator; icons are retrieved from the Noun Project Icons API.Source: §F.2 · p. 21, Figure 18 · p. 22
  • GPT-4Vision does not improve plan creation over text-only GPT-4 and tends to overestimate diagram correctness during refinement.In examples, text-only GPT-4 identifies some errors that GPT-4Vision says do not require improvement.Source: §F.2 · pp. 21–22, Figure 19 · p. 23, Figure 20 · p. 23
  • TextDiffuser-2 can produce good labels but sometimes merges letters, whereas Pillow guarantees error-free rendering and supports font adjustments.The modular framework can incorporate a text-rendering model if the end user wants it; the comparison includes TextDiffuser-2 (Chen et al., 2023b).Source: §F.2 · pp. 22–23, Figure 21 · p. 24
  • GPT-4 generates diverse diagram plans across prompts and domains, including different plans for the same prompt.Figure 17 illustrates variation between runs for one prompt.Source: Figure 17 · p. 22, §F.2 · p. 23

F.3 Ablation Studies

  • More denoising steps with layout guidance improve count performance but can reduce object realism; the prose selects α=10 to balance accuracy and relationships.Smaller α favors object generation, while larger α strengthens layout control and count performance.Source: §F.3 · p. 24, Table 5 · p. 24
  • Full-model fine-tuning improves performance and visual quality, while ground-truth plans outperform GPT-4 plans, which score close to the oracle.The authors use the fully fine-tuned model as their default and use ground-truth plans to measure DiagramGLIGEN’s upper bound and assess room for planner improvement.Source: Table 6 · p. 24, §F.3 · p. 25, Table 7 · p. 25

G Limitations

DiagrammerGPT can support educational and human-in-the-loop diagram work, but it can make errors, be used to create misleading diagrams, and incur costly LLM inference. Its generator also inherits limitations from GLIGEN and Stable Diffusion v1.4.

G Limitations

  • The framework can make errors or create false or misleading diagrams, so real-world use should include human supervision.The paper points to human-in-the-loop diagram-plan editing as an example of such supervision.Source: G Limitations · p. 25
  • Generating plans with the strongest LLM APIs can be costly, though quantization, distillation, and open-source models may lower inference costs.Source: G Limitations · p. 25
  • Because DiagramGLIGEN uses pretrained GLIGEN and Stable Diffusion v1.4 weights, it shares their training-distribution deviations, imperfect generation quality, and English-only understanding.Source: G Limitations · p. 25

Related papers

Loading 2310.12128v2…