PaperlayerView exact revision on arXiv

Source-linked AI summary

Empowering LLMs to Understand and Generate Complex Vector Graphics

Ximing Xing, Juncheng Hu, Guotao Liang, Jing Zhang, Dong Xu, Qian Yu

arXiv:2412.11102v3cs.CV

TL;DR

LLMs have yet to fully address complex SVG generation, while SVG instruction data is scarce. LLM4SVG combines semantic SVG tokens, an instruction dataset, and LLM fine-tuning; the authors report a framework for SVG understanding and generation, with a stated scope condition on suitable base models.

  • Problem

    Complex SVG generation remains difficult for LLMs, and large-scale vector-graphics instruction-following data is scarce.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    LLM4SVG adapts LLMs with learnable semantic tokens and an SVG instruction dataset built from human-designed graphics and automated data generation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    LLM4SVG is presented as a framework for understanding SVG source code and generating high-quality SVGs with meaningful complexity.

    Source: §6 · p. 8

  • Takeaways & Limitations

    SVGX-SFT provides 580k instruction-following samples intended to support LLM training for vector graphics.

    Source: §1 · p. 2, §3 · p. 4

    The framework’s base models need capabilities for both visual and textual data and effective instruction following; the authors suggest other similar models could also be used.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

The unprecedented advancements in Large Language Models (LLMs) have profoundly impacted natural language processing but have yet to fully embrace the realm of scalable vector graphics (SVG) generation. While LLMs encode partial knowledge of SVG data from web pages during training, recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions. Additionally, LLM training typically lacks modeling and understanding of the rendering sequence of vector paths, which can lead to occlusion between output vector primitives. In this paper, we present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics. LLM4SVG facilitates a deeper understanding of SVG components through learnable semantic tokens, which precisely encode these tokens and their corresponding properties to generate semantically aligned SVG outputs. Using a series of learnable semantic tokens, a structured dataset for instruction following is developed to support comprehension and generation across two primary tasks. Our method introduces a modular architecture to existing large language models, integrating semantic tags, vector instruction encoders, fine-tuned commands, and powerful LLMs to tightly combine geometric, appearance, and language information. To overcome the scarcity of SVG-text instruction data, we developed an automated data generation pipeline that collected our SVGX-SFT Dataset, consisting of high-quality human-designed SVGs and 580k SVG instruction following data specifically crafted for LLM training, which facilitated the adoption of the supervised fine-tuning strategy popular in LLM development.

1. Introduction

Existing SVG-generation approaches often handle only simplified graphics or convert raster images into SVGs that remain noneditable. LLMs show promise with simple primitives but can struggle with complex SVG structure and attributes; LLM4SVG addresses this gap with semantic tokens, a modular architecture, and a large instruction dataset.

Motivation

  • Current approaches either restrict SVGs to basic path commands or rasterize images before conversion, leaving a gap in direct synthesis of intricate, detailed, editable SVG code.The paper notes that raster-to-SVG methods involve cumbersome iteration and that their outputs do not meet professional designers’ expectations.Source: §1 · p. 2
  • LLMs can generate simple primitives but may hallucinate vector paths and improperly encode attributes when SVG code is embedded among lengthy XML and web-page content.The paper attributes these difficulties to SVG data being less accessible during pretraining because models must parse layered languages.Source: §1 · p. 2

Contributions

  • LLM4SVG uses learnable semantic tokens to encode SVG components and properties, alongside a modular architecture that combines geometry, appearance, and language information.The framework is presented as supporting existing LLMs and MLLMs for both SVG understanding and generation.Source: §1 · p. 2
  • SVGX-SFT combines human-designed SVGs with 580k instruction-following examples tailored to LLM training.The dataset is introduced to support the paper’s supervised fine-tuning strategy.Source: §1 · p. 2

2. Related Work

Prior work explores command-based SVG generation, raster-to-vector methods, vector recognition, and language-model adaptation, but the paper identifies gaps in complexity, structured understanding, and bidirectional language–graphics workflows. LLM4SVG follows an end-to-end fine-tuning strategy for pre-trained LLMs.

2.1. Vector Graphics Generation

  • Command-based neural generators rely on vector datasets and often face limited generalization to intricate graphics, while raster-guided methods optimize SVG path geometry and color or combine diffusion with differentiable rasterizers.The paper also describes methods that combine optimization with neural networks to learn vector representations under geometric constraints.Source: §2.1 · pp. 2–3

2.2. Vector Graphics Understanding

  • Recognition systems and benchmarks have advanced, but the paper says existing approaches do not adequately address representation issues or the bidirectional language–graphics relationship needed for creative workflows.The cited discussion distinguishes recognition models lacking generation from evaluation frameworks that do not address fundamental representation issues.Source: §2.2 · p. 3

2.3. Large Language Models

  • LLM adaptation can use modality-specific interfaces or end-to-end training, and this work follows the latter by fine-tuning pre-trained LLMs for vector-graphics understanding and generation.The paper presents this strategy as retaining flexibility for multimodal extensions.Source: §2.3 · p. 3

2.4. Instruction Tuning

  • Instruction tuning has been used to improve language and multimodal task performance, while some SVG methods fine-tune LLMs on image embeddings but treat structured SVG code as sequential text.The paper characterizes those SVG approaches as overlooking the hierarchical and structured nature of SVG files.Source: §2.4 · p. 3

3. SVGX-SFT Dataset

The paper builds SVGX-SFT to address the scarcity and production cost of large-scale vector-graphics instruction data. It normalizes collected SVGs, generates image captions and instruction-following examples, and organizes examples for both generation and understanding.

Dataset motivation

  • Large-scale instruction-following data is difficult to acquire for vector graphics, contributing to research focused on a limited set of applications such as hand drawings, fonts, and icons.The paper describes production costs and tagging challenges as constraints on the field’s scope.Source: §3 · p. 3

Data collection and preprocessing

  • The dataset begins with approximately 250,000 colorful, complex vector graphics, normalized for consistent canvas size, relative coordinates, and representation.A preprocessing pipeline losslessly reduces SVG files by removing redundant content before they are rasterized at 512 × 512 pixels and captioned with BLIP.Source: §3 · pp. 3–4

Instruction templates

  • Five instruction templates divide tasks into text- or text-and-image-to-SVG generation and three forms of SVG understanding, with loss computed only on model responses.The templates use BLIP-generated prompts, GPT-4-generated descriptions, and semantic tokens; Path_i represents a complete SVG primitive.Source: Table 1 · p. 4

Instruction data

  • Text-SVG pairs support text-to-SVG generation, text-image-SVG triples support generation from text and visual prompts, and other templates target graphics comprehension.The paper groups templates #1–#2 under synthesis and #3–#5 under comprehension.Source: §3 · p. 4

Dataset analysis

  • Figure 2 summarizes frequent dataset words, element-count changes after cleaning, names and descriptions, and the distribution of SVG group elements.The caption says cleaning reduces element counts, while the accompanying analysis notes that path elements and coordinates decrease and rect, circle, and ellipse counts remain relatively stable.Source: Figure 2 · p. 4

Instruction data

  • The authors used ChatGPT and GPT-4 to generate instruction-following data because BLIP captions were suitable as prompts but too short for comments intended to support SVG understanding.The captions were generated from rendering results during the re-captioning phase.Source: §3 · p. 4

Dataset composition

  • SVGX-SFT includes 250k annotated, high-quality, standardized vector graphics and 580k unique SVG-Text-Image samples, with the passage additionally reporting a distribution across five sample types.Source: §3 · p. 4

4. Instruction Tuning Scheme

LLM4SVG converts SVG code into a structured token representation and trains an LLM to understand and generate graphics from instructions. Its scheme combines token initialization, optional visual input, and staged training, with base-model choice constrained by the models’ multimodal and instruction-following capabilities.

4.1. SVG Semantic Tokens

  • The overview separates understanding, which predicts descriptions from SVG code, from generation, which predicts masked SVG code; rendered images may also condition either task.For understanding training, detailed GPT-4 descriptions serve as labels; at inference, the model describes SVG source code or generates SVG from a text prompt.Source: Figure 3 · p. 5
  • LLM4SVG introduces 55 semantic tokens—15 tag, 30 attribute, and 10 path-command tokens—to replace literal SVG tags and attributes in source code.This representation is intended to preserve SVG-token uniqueness and align tokens with SVG definitions.Source: §4.1 · p. 5
  • New SVG-token embeddings are initialized from the semantic average of the tokens in each token’s descriptive text.The paper says this initialization provides a starting point and a compact distributed representation for SVG tokens.Source: §4.1 · p. 5

4.2. Architecture

  • The architecture uses GPT-2, Phi-2, or Falcon as its base LLM, and the authors say other models could serve if they support visual and textual data and instruction following.These base models are denoted fϕ in the paper.Source: §4.2 · p. 5

4.3. Training

  • Training samples multi-turn conversations from the SVG instruction dataset, with X_un and X_gen denoting understanding and generation tasks and T the total number of turns.The dataset instructions provide the task-specific inputs and responses.Source: §4.3 · p. 5
  • The training objective maximizes the likelihood of target answer tokens autoregressively, conditioned on SVG, instructions, preceding answers, and previously predicted tokens.The model predicts a sequence of length L; θ denotes trainable parameters, while X_inst and X_a represent instruction and answer tokens from preceding turns.Source: §4.3 · p. 5
  • Stage 1 aligns features by training only the word embeddings while keeping the visual encoder and LLM frozen.The paper converts paired samples to instruction-following data and randomly samples instructions for this stage.Source: §4.3 · pp. 5–6
  • Stage 2 fine-tunes the full instruction dataset end-to-end, either with parameter-efficient methods such as LoRA or by updating all model parameters.The paper notes that full fine-tuning requires greater computational resources, especially for large LLMs.Source: §4.3 · p. 6

5. Experiments

The experiments evaluate LLM4SVG’s training setup, dataset, generation quality, and architecture. They report stronger performance than other LLM-based methods while noting differences from optimization-based methods and examining model and token design.

5. Experiments

  • Training adds 55 SVG semantic tokens, then applies supervised fine-tuning with LoRA/QLoRA or full parameters; the reported run uses two epochs and evaluates visual quality, alignment, aesthetics, preference, and cost.The implementation uses AdamW, a cosine learning-rate schedule, a 4096-token maximum, and eight NVIDIA A800 GPUs.Source: §5 · p. 6

5.1. SVGX-SFT Dataset Analysis

  • 23.1% of SVGs have five groups; the dataset is UI-oriented, cleaning reduces path elements and coordinates, and group counts indicate hierarchical structure.Common terms include “logo,” “document,” “button,” and “download”; <rect>, <circle>, and <ellipse> remain relatively stable after cleaning.Source: §5.1 · p. 6

5.2. Quantitative Evaluation

  • LLM4SVG achieves the highest performance among LLM-based SVG generation methods, remains competitive across several metrics, and generates significantly faster than optimization-based methods.The comparison covers visual quality and computational cost; LLM4SVG requires inference rather than a time-consuming optimization process.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • LLM4SVG outperforms all other LLMs across every aspect in the Table 2b comparison.The method incorporates decimal coordinates and hexadecimal color codes for SVG paths to represent a wider and more accurate range of colors.Source: §5.2 · p. 7

5.3. Qualitative Evaluation

  • LLM4SVG produces more complete graphics with better primitive selection, placement, and semantic richness than other LLM-based methods, while optimization methods can use redundant, irregular primitives.The user study found LLM4SVG significantly outperformed other LLM-based methods, with visual quality closely comparable to human-designed SVGs.Source: §5.3 · pp. 7–8, Figure 4 · p. 7, Table 3 · p. 8

5.4. Ablation Study

  • Base-model tokenization affects coordinate and color generation, while description-initialized semantic-token embeddings form structured clusters and random initialization scatters.The paper contrasts Qwen number tokenization with GPT-2’s individually enumerated numbers; randomly initialized tokens are associated with slower convergence.Source: §5.4 · p. 8

6. Conclusion & Discussion

The conclusion presents LLM4SVG as a framework for existing language models to understand and generate SVGs, supported by structured encoding and a human-designed SVG instruction dataset. It outlines future work in semantic editing, cross-modal translation, iterative refinement, and domain adaptation.

6. Conclusion & Discussion

  • LLM4SVG is presented as the first framework designed to support existing LLMs and MLLMs in both understanding and generating SVGs.Its structured SVG encoding is intended to address treating SVG source code merely as plain text.Source: §6 · p. 8
  • The SVGX-SFT Dataset combines high-quality human-designed SVGs with dialogue-based instruction pairs tailored for training LLMs.The authors anticipate that the dataset will accelerate future vector-graphics research.Source: §6 · p. 8
  • Future directions include semantic SVG editing, text–raster–vector translation, natural-language iterative refinement, and adaptation to domains such as data visualization and UI/UX design.Source: §6 · p. 8

Overview

The supplementary material describes comparisons, data preparation, semantic tokens, user evaluation, primitive ordering, and additional generated examples. It also outlines alternative composition strategies and the evaluation procedures.

Overview

  • Supplementary sections cover comparisons with existing LLM-based methods, dataset collection and lossless preprocessing, and SVG semantic tokens with embedding-initialization analysis.Source: Overview · p. 12
  • The material adds user-study methodology and compares step-by-step composition with top-down refinement for primitive ordering and SVG interpretability.Source: Overview · p. 12
  • Additional examples showcase generated SVGs across objects, animals, food, and abstract symbols, with claimed semantic accuracy and stylistic consistency.Source: Overview · p. 12

A. Comparison with LLM-based Methods

LLM4SVG produces more complete, detailed, and prompt-aligned graphics than the compared LLM-based methods. Its decimal coordinates also avoid distortions and incompletion associated with integer-only coordinates.

A. Comparison with LLM-based Methods

  • LLM4SVG produces more complete graphics, more diverse shapes, more harmonious colors, and closer prompt alignment than the compared LLMs.The comparisons note that other models can produce incomplete graphics, overly simplistic shapes, or colors that lack harmony.Source: A. Comparison with LLM-based Methods · p. 12
  • Integer-only coordinates can distort or leave SVG shapes incomplete, whereas LLM4SVG generates decimal coordinates accurate to two decimal places.The umbrella comparison contrasts LLM4SVG with GPT-o1-preview and Claude 3.5-sonnet.Source: A. Comparison with LLM-based Methods · p. 12, Figure S1 · p. 12

B. SVGX-SFT Dataset Details

SVGX-SFT combines diverse SVG examples with preprocessing and instruction construction for generation and understanding. LLM4SVG represents SVG elements with dedicated semantic tokens, initialized using their descriptions.

B. SVGX-SFT Dataset Details

  • SVGX-SFT spans primitives from minimal to highly detailed and covers people, animals, objects, and symbols.Illustrations also vary in style, structure, and semantics.Source: B. SVGX-SFT Dataset Details · p. 12, Figure S2 · p. 13
  • Preprocessing removes redundant SVG content, converts absolute coordinates to relative ones, rounds decimals to two places, and standardizes canvases at 128×128.The pipeline is described as preserving visual fidelity while reducing file size and computational overhead; understanding samples pair rasterized images with GPT-4 descriptions, while generation samples pair BLIP prompts with SVG source code.Source: B. SVGX-SFT Dataset Details · p. 13
  • The dataset contains 580k high-quality SVG-text instruction-compliant samples for structured vector graphic generation and interpretation.Source: B. SVGX-SFT Dataset Details · p. 13

C. Our Proposed SVG Semantic Tokens

  • LLM4SVG uses 15 tag, 30 attribute, and 10 path-command tokens to replace SVG tags and attributes, with descriptions used to initialize the tokens.Description-based initialization yields a more compact embedding region than initialization without descriptions, and training groups tokens with similar semantics.Source: C. Our Proposed SVG Semantic Tokens · p. 13

D. More Details about User Study

The supplementary section details a user evaluation protocol and illustrates LLM4SVG’s SVG comparisons and processing. It also describes primitive ordering that progresses from foundational elements to a complete design, alongside a top-down alternative.

D. More Details about User Study

  • The user study sampled 17 prompts, yielding 68 SVGs, and 37 volunteers rated prompt alignment, visual quality, and practical usability.Questionnaires showed five randomly sampled SVGs; volunteers completed one to three questionnaires, with alignment and quality rated on a 1–5 scale and usability recorded yes/no.Source: D. More Details about User Study · pp. 13–14, Figure S7 · p. 17
  • The qualitative comparison reports that LLM4SVG improves coherence, accuracy, and stylistic refinement over existing LLM-based methods.Source: Figure S3 · p. 14

E. Primitive Ordering in SVG Generation

  • The SVG cleaning pipeline removes redundant metadata, converts absolute coordinates to relative ones, standardizes canvas size, and preserves the rendered appearance.Source: Figure S5 · p. 15
  • Table S1 defines 15 tag tokens, 30 attribute tokens, and 10 path-command tokens in LLM4SVG.Source: Table S1 · p. 16
  • Figure S8 contrasts incremental construction from basic primitives with a top-down sequence that introduces the whole object before refinements.The incremental sequence is described as resembling human sketching, while the alternative lacks that structured progression.Source: E. Primitive Ordering in SVG Generation · pp. 14, 17, Figure S8 · p. 17
  • LLM4SVG’s ordering is intended to make graphics more interpretable and usable for step-by-step drawing, vector editing, and automated graphic generation.Source: E. Primitive Ordering in SVG Generation · p. 17

F. Additional Results Generated by Our LLM4SVG

LLM4SVG generates illustrations across a broad range of subjects, with the examples described as visually coherent and stylistically consistent. The collection includes objects, animals, food, people, symbols, and abstract concepts.

F. Additional Results Generated by Our LLM4SVG

Loading 2412.11102v3…