PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

Abhay Zala, Han Lin, Jaemin Cho, Mohit Bansal

arXiv:2310.12128v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Text-to-image models struggle with diagrams because they lack precise control over dense layouts and readable labels. DiagrammerGPT uses LLM-generated, iteratively refined plans followed by DiagramGLIGEN rendering and explicit label synthesis, and it produces more accurate diagrams than existing baselines.

  • Problem

    Existing T2I models often fail to generate diagrams with correct dense object relationships, fine-grained layouts, and comprehensible text labels.

  • Method

    DiagrammerGPT uses an LLM planner-auditor loop to create and refine diagram plans, then DiagramGLIGEN and explicit text rendering to generate diagrams from them.

  • Results

    DiagrammerGPT generates more accurate diagrams than existing T2I and diagram-generation baselines in quantitative, qualitative, and human preference evaluations.

  • Takeaways & Limitations

    The framework supports open-domain generation, multiple vector-graphics platforms, human-in-the-loop editing, and multimodal planner or auditor LLMs.

  • Takeaways & Limitations

    DiagramGLIGEN can still make generation mistakes, while real-world use requires human supervision and LLM inference can be costly.

Abstract

from arXiv · show

Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. A diagram is a symbolic/schematic representation that explains information using structurally rich and spatially complex visualizations (e.g., a dense combination of related objects, text labels, directional arrows/lines, etc.). Existing state-of-the-art T2I models often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows/lines, and also often fail to render comprehensible text labels. To address this gap, we present DiagrammerGPT, a novel two-stage text-to-diagram generation framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams. In the first stage, we use LLMs to generate and iteratively refine 'diagram plans' (in a planner-auditor feedback loop). In the second stage, we use a diagram generator, DiagramGLIGEN, and a text label rendering module to generate diagrams (with clear text labels) following the diagram plans. To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset. We show that our DiagrammerGPT framework produces more accurate diagrams, outperforming existing T2I models. We also provide comprehensive analysis, including open-domain diagram generation, multi-platform vector graphic diagram generation, human-in-the-loop editing, and multimodal planner/auditor LLMs.

1 Introduction

DiagrammerGPT addresses the difficulty of generating accurate diagrams by separating LLM-based layout planning from layout-guided visual generation and explicit text rendering. It outperforms existing baselines in diagram accuracy while supporting open-domain generation and multiple output platforms.

  • Diagram planning: The framework first uses an LLM planner and auditor feedback loop to generate and refine diagram plans containing entities, relationships, and layouts.The auditor identifies errors such as incorrect object positions or relationships, and the planner updates the plan.
  • Diagram generation: DiagramGLIGEN then follows the refined plans to generate diagram objects, while explicit text rendering improves label readability.DiagramGLIGEN is specialized for diagrams with text labels and arrows as additional layout inputs.
  • Dataset: AI2D-Caption supplies detailed captions and object descriptions for training and benchmarking text-to-diagram generation.The dataset is built on AI2D and annotated with LLaVA 1.5.
  • Results: DiagrammerGPT outperforms existing text-to-image and diagram-generation baselines in diagram accuracy, including human preference for image-text alignment and object relationships.The comparison includes Stable Diffusion v1.4, VPGen, and AutomaTikZ in zero-shot and fine-tuned settings.
  • Scope: Additional analyses examine unseen domains, PowerPoint/Inkscape/Illustrator rendering, human-in-the-loop editing, and multimodal planner or auditor LLMs.The framework often produces semantically accurate plans and diagrams for unseen domains such as geology and plants.

2 Related Works

Related work establishes that modern T2I systems produce photorealistic images and that LLMs can guide layouts, but diagram generation remains difficult because diagrams require precise relational structure and readable labels.

  • Diagram generation: DiagrammerGPT's second stage combines layout-guided object generation with Pillow-based text rendering to address these diagram-specific requirements.Figure 3 depicts DiagramGLIGEN generating objects from plans before clear labels are rendered.
  • Text-to-image generation: Existing T2I models often lack fine-grained layout control for densely connected diagram objects and frequently generate illegible text labels.These limitations are especially problematic because diagrams prioritize correct information and object relationships over photorealism.
  • LLM-guided generation: Prior LLM-guided layout methods focus on controlling natural-image generation rather than generating diagrams.The related approaches leverage LLMs for downstream image layout control but do not provide diagram-specific capabilities.

3 DiagrammerGPT: Method Details

DiagrammerGPT converts text prompts into diagrams through two stages: an LLM creates and refines a structured diagram plan, then DiagramGLIGEN renders the planned objects and labels.

  • 3.1 Stage 1: Diagram Planning: GPT-4 initially generates plans using ten in-context examples, with entities, complex relationships, and bounding-box layouts as core components.The plan format supports object-object and object-text-label relationships.
  • 3.1 Stage 1: Diagram Planning: An auditor LLM checks the plan against the prompt and feeds detected omissions or arrangement errors back to the planner for iterative refinement.The planner and auditor form a feedback loop intended to improve initially generated plans.
  • 3.2 Stage 2: Diagram Generation: DiagramGLIGEN generates diagram images from the plan, after which text labels are explicitly rendered for clarity.The generation stage separates object creation from label rendering.
  • 3.2 Stage 2: Diagram Generation: DiagramGLIGEN is a layout-guided diagram model designed to address omitted objects, incorrect relationships, and unreadable labels in existing T2I systems.It adapts GLIGEN for the diagram domain and uses layout guidance to prioritize factual structure over photorealism.

4 Experimental Setup

The evaluation uses AI2D-Caption, multiple text-to-image baselines, automated diagram-quality metrics, and human assessments of pipeline stages and model preferences.

  • 4.1 AI2D-Caption Dataset: AI2D-Caption extends about 4.9K AI2 diagrams with detailed diagram captions and region descriptions for bounding boxes.The annotations address the original dataset’s short or missing object descriptions.
  • 4.2 Baselines: DiagrammerGPT is compared with Stable Diffusion v1.4, VPGen, and AutomaTikZ using zero-shot and applicable AI2D-Caption fine-tuned settings.VPGen is fine-tuned in both its Vicuna13B and GLIGEN components.
  • 4.3 Evaluation Metrics: Evaluation combines VPEval, captioning with CIDEr and BERTScore, and CLIPScore comparisons against captions and ground-truth diagrams.VPEval measures object presence, object counts, relationships, and text labels; CLIPScore uses image-text and image-image variants.
  • 4.3 Evaluation Metrics: Table 1 reports comparisons between DiagrammerGPT and existing text-to-image baseline models across the evaluation metrics.The table caption states that DiagrammerGPT outperforms the baselines on all metrics.
  • 4.4 Human Evaluation: Human evaluation rates object presence and relationships across both pipeline stages on 25 layouts or images using a 1-to-5 Likert scale.A separate pairwise study compares DiagrammerGPT with fine-tuned Stable Diffusion v1.4 on 50 diagrams using 20 crowd-sourced annotators.

5 Results and Discussion

DiagrammerGPT outperforms baseline text-to-image and diagram-generation methods on diagram accuracy, while supporting open-domain, vector-graphic, and human-editable outputs. Its main remaining bottleneck is the DiagramGLIGEN generation backbone, which can fail to follow otherwise-correct plans.

  • 5.1 Quantitative Results: DiagrammerGPT outperforms zeroshot and fine-tuned baselines on overall and skill-specific VPEval scores, including object relationships and text rendering.Fine-tuning improves baseline object and relationship scores but leaves both baselines at zero on text; DiagrammerGPT surpasses both settings overall and by skill.
  • 5.1 Quantitative Results: 26.4 vs. 4.2 and 18.2 CIDEr scores show DiagrammerGPT outperforming fine-tuned VPGen and Stable Diffusion v1.4, respectively.It also improves over fine-tuned Stable Diffusion v1.4 on both reported CLIPScores: 32.1 vs. 30.1 and 73.9 vs. 68.1.
  • 5.1 Quantitative Results: Refinement raises object-relationship scores from 4.56 to 4.72, while final generation lowers object-presence and relationship scores relative to the plans.The step-wise analysis evaluates 25 AI2D-Caption prompts using a 1–5 Likert scale.
  • 5.2 Human Evaluation: Human evaluation finds DiagrammerGPT preferable to Stable Diffusion v1.4 on image-text alignment and object relationships.The comparison is reported as a pairwise preference study.
  • 5.3 Qualitative Analysis: The method’s generated plans are generally correct, but DiagramGLIGEN can fail to follow all plan aspects and may improve with a stronger backbone than Stable Diffusion v1.4.The qualitative comparison shows correct four-phase Earth layouts for DiagrammerGPT while baselines over-generate objects or fail to produce a proper diagram.
  • 5.4 Additional Analysis: DiagrammerGPT generates semantically accurate layouts in unseen domains, vector-graphic platforms, and editable human-in-the-loop workflows.Users can move, add, remove, or resize objects before retaining the vector plan or regenerating the diagram.
  • 5.5 Different LLMs for diagram plan generation: GPT-4 achieves a 78.4 overall planner score and slightly exceeds GPT-3.5 Turbo on object and text metrics, motivating its use as the main LLM.LLaMA3 and fine-tuned Vicuna each score below 30 overall.

6 Conclusion

The paper proposes DiagrammerGPT, a two-stage framework that uses LLMs to plan and refine diagrams before generating them. It reports more semantically accurate layouts than baselines and extends the workflow to broader generation, editing, and platform settings.

  • 6 Conclusion: DiagrammerGPT uses LLM knowledge to plan and refine diagram layouts before diagram generation.The framework is presented as a two-stage text-to-diagram generation approach.
  • 6 Conclusion: The framework achieves more semantically accurate layouts than baseline models in quantitative and qualitative analyses.The paper also reports human error analysis and ablation studies.
  • 6 Conclusion: The paper analyzes open-domain generation, vector-graphic diagrams, human-in-the-loop plan editing, and multimodal planner/auditor LLMs.These analyses broaden the demonstrated uses of the diagram-planning framework.
  • 6 Conclusion: The appendix supplies prompt templates, dataset-collection details, DiagramGLIGEN and experimental setups, human-evaluation procedures, additional analyses, and limitations.The planner and auditor prompts use ten in-context examples, while GPT-4 diagram-plan generation costs $0.17 on average.

B AI2D-Caption Collection Details

AI2D-Caption adapts AI2D for text-to-diagram generation by adding captions and region descriptions that support planning and layout-guided generation. The annotations are produced with LLaVA 1.5 and checked through human evaluation.

  • B AI2D-Caption Collection Details: AI2D-Caption adds diagram captions and object descriptions because AI2D titles are often too short and its boxes are labeled only as “blob.”The dataset is built from AI2D annotations containing titles, bounding boxes, and object linkages.
  • B AI2D-Caption Collection Details: The planner prompt asks for entities, relationships, and [x, y, width, height] locations normalized from the top-left coordinate system.Entities may be images or text, and the instructions decompose each caption into parts.
  • B AI2D-Caption Collection Details: The auditor prompt checks whether a proposed diagram layout is wrong given its caption, entities, relationships, and locations.It requests a step-by-step explanation of why the diagram is correct or incorrect.
  • B AI2D-Caption Collection Details: LLaVA 1.5 generates captions by answering what each diagram shows and describes each overlaid bounding-box region.Boxes receive labels such as B1 before region-description prompting.
  • B AI2D-Caption Collection Details: Human evaluation finds LLaVA captions very good 80% of the time and bounding-box descriptions very good 68% of the time.Caption errors are usually minor, while region errors can assign nearby boxes the same description.
  • B AI2D-Caption Collection Details: DiagramGLIGEN is initialized from GLIGEN’s Box+Text checkpoint and trained on AI2D-Caption for 15k steps.Training uses batch size 5 per GPU and takes 12 hours on eight A6000 GPUs.

D.1 Metrics

VPEval assesses diagrams across object presence, object counts, spatial and connection relationships, and text-label correctness using specialized evaluation modules.

  • VPEval measures object presence, object counts, spatial and connection relationships, and correct text labels.Its programs invoke modules such as Object, OCR, and VQA to evaluate different image aspects.
  • Ground-truth diagram plans determine evaluation programs for objects, counts, and text, while GPT-4-generated VQA questions assess relationships.Relationship questions test spatial or connection properties such as whether the moon lies between the sun and Earth.
  • For count evaluation, LLaVA checks whether the diagram contains exactly N instances of each required object.The same model determines object presence and whether spatial or connection relations are true.
  • Because LLaVA often produces false positives for relationship evaluation, the authors apply an additional filtering step.

E Human Evaluation Setup Details

The human preference study uses qualified English-speaking crowd-workers to compare diagrams on object relationships and alignment with the input prompt.

  • The study recruits AMT Masters workers with more than 1000 completed HITs and approval ratings above 95%.Workers come from the United States, Great Britain, Australia, or Canada because the task is in English.
  • Workers are paid $0.06 per diagram pair, corresponding to roughly $14-15 per hour.
  • Figure 12 shows the interface used during human evaluation.
  • Annotators compare which diagram better captures object relationships and better represents the input sentence.Object relationships include spacing, positioning, and arrows or lines between objects.

F.1 Qualitative Results and Analysis

Qualitative examples show that DiagrammerGPT more closely follows prompts and diagram plans than baseline systems, while refinement improves connections and prevents overlaps.

  • Qualitative comparison to baselines: DiagrammerGPT produces plans and final diagrams that better reflect prompts than Stable Diffusion v1.4 and AutomaTikZ.Stable Diffusion over- or under-generates objects, while AutomaTikZ fails to produce proper layouts and objects.
  • Qualitative comparison to baselines: In examples, DiagrammerGPT correctly represents the four phases of Earth revolving around the Sun and the circular flow of a butterfly life cycle.Its generated diagrams capture most aspects of the planned life cycle, although some individual stages can still be misdrawn or swapped.
  • Qualitative comparison to baselines: DiagrammerGPT can still fail to follow every planned detail, including misdrawing the egg stage or swapping larva and pupa.The authors note that a stronger DiagramGLIGEN backbone could improve adherence to plans.
  • Diagram plan refinement: The auditor refinement step corrects missing circuit connections and rearranges layouts for clearer flow.In the circuit example, refinement connects the switch to the battery so it affects the circuit.
  • Diagram plan refinement: Refinement removes an overlap between the New Moon label or object and the Earth object.

F.2 Additional Analysis

Additional analyses examine open-domain generation, platform export, planner diversity, multimodal planning, text rendering, and DiagramGLIGEN design choices.

  • Open-domain diagram generation: DiagrammerGPT generates semantically accurate plans and diagrams in unseen domains, although DiagramGLIGEN can struggle in some cases.The planner generalizes to domains such as geology, botany, neural networks, and vacation planning, while the generated diagrams can strongly adhere to plans.
  • Open-domain diagram generation: DALL-E 3 generally offers better aesthetic style but produces redundant, crowded objects and diagrams that less accurately follow prompts.Examples include incorrect Earth layers, excessive text, too many plants, and plants arranged in the wrong number of columns.
  • Vector graphic diagram generation in different platforms: Diagram plans can be rendered as vector graphics in Microsoft PowerPoint, Inkscape, and Adobe Illustrator.The implementations use VBA, a Python extension, and JavaScript respectively, with icons retrieved from the Noun Project Icons API.
  • GPT-4 vs. GPT-4Vision: GPT-4Vision does not produce better plans than text-only GPT-4 and tends to overestimate correctness during refinement.Text-only GPT-4 can identify some errors that GPT-4Vision judges not to require improvement.
  • Text rendering: TextDiffuser-2 can generate good labels but may merge letters, whereas Pillow guarantees error-free rendering and supports font adjustments.The modular system can incorporate a model-based renderer if desired.
  • Planner diversity: GPT-4 generates diverse diagram plans across prompts and across repeated runs with the same prompt.The examples indicate diversity across dense, open-domain, and multiple application settings.
  • DiagramGLIGEN ablations: Increasing layout-guided denoising steps improves count performance but can reduce object-generation quality and realism.The authors select α as 10 steps for a balance across object, count, and relationship accuracy, while the table caption reports α = 15 as the default setting.
  • DiagramGLIGEN ablations: Full fine-tuning improves DiagramGLIGEN performance and visual quality, so it is used as the default model.Using ground-truth plans performs better than GPT-4 plans, although GPT-4 comes close to the oracle score.

G Limitations

DiagrammerGPT remains constrained by generation quality, language coverage, cost, and the need for human supervision in real-world use.

  • G Limitations: Human supervision is recommended because the framework can produce errors or misleading diagrams and may be used to create false information.Strongest LLM APIs can also make diagram-plan generation costly.
  • G Limitations: DiagramGLIGEN inherits limitations from GLIGEN and Stable Diffusion v1.4, including training-distribution deviations, imperfect generation quality, and English-only understanding.
Loading 2310.12128v2…