PaperlayerView exact revision on arXiv

Source-linked AI summary

StarVector: Generating Scalable Vector Graphics Code from Images and Text

Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli

arXiv:2312.11556v4cs.CVcs.AIcs.CL

TL;DR

Existing SVG-generation methods can lack semantic understanding and broad primitive use, while pixel-based metrics may not capture vector quality. StarVector generates SVG code from image or text inputs, supported by SVG-Stack and SVG-Bench. The paper reports stronger visual quality and primitive use than baselines, while noting constraints in context length, visual feedback, and generation speed.

  • Problem

    Existing SVG-generation methods often rely on curves, lack semantic understanding, and struggle to use SVG primitives across diverse tasks.

    Source: §1 · pp. 1–2

  • Method

    StarVector is a multimodal language model that generates SVG code from images or text, trained using SVG-Stack and evaluated with SVG-Bench and SVG-specific metrics.

    Source: §1 · p. 2, §4.1 · p. 5, §5.1 · p. 5

  • Results

    The paper reports superior visual quality, precise line and shape rendering, and primitive use versus baselines, with StarVector leading DinoScore on six of eight image-vectorization benchmarks.

    Source: §6.1 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The study finds pixel-based metrics inadequate for capturing SVG quality and proposes perceptual alternatives alongside a benchmark spanning 10 datasets and three tasks.

    Source: §1 · p. 2, §7 · p. 8

    StarVector’s 16k-token context is inadequate for complex SVGs; it also relies on minimal visual feedback and is limited by LLM generation speed.

    Source: §7 · p. 8

Abstract

from arXiv · show

Scalable Vector Graphics (SVGs) are vital for modern image rendering due to their scalability and versatility. Previous SVG generation methods have focused on curve-based vectorization, lacking semantic understanding, often producing artifacts, and struggling with SVG primitives beyond path curves. To address these issues, we introduce StarVector, a multimodal large language model for SVG generation. It performs image vectorization by understanding image semantics and using SVG primitives for compact, precise outputs. Unlike traditional methods, StarVector works directly in the SVG code space, leveraging visual understanding to apply accurate SVG primitives. To train StarVector, we create SVG-Stack, a diverse dataset of 2M samples that enables generalization across vectorization tasks and precise use of primitives like ellipses, polygons, and text. We address challenges in SVG evaluation, showing that pixel-based metrics like MSE fail to capture the unique qualities of vector graphics. We introduce SVG-Bench, a benchmark across 10 datasets, and 3 tasks: Image-to-SVG, Text-to-SVG generation, and diagram generation. Using this setup, StarVector achieves state-of-the-art performance, producing more compact and semantically rich SVGs.

1. Introduction

SVG vectorization must generalize across image types while preserving semantic content and choosing appropriate SVG primitives. StarVector generates SVG code from images or text, and the paper introduces a dataset and benchmark to support training and evaluation.

1. Introduction

  • Existing methods struggle to generalize across fonts, logos, illustrations, and diagrams, while often underusing SVG primitives and lacking semantic understanding [7] [8] [69] [70].Path-based approximations can represent a circle less compactly than a <circle/> primitive, and text as paths loses editable text content.Source: §1 · pp. 1–2
  • For the planet example, StarVector’s MSE is 0.009 versus LIVE’s 0.0012 and VTracer’s 0.0039, despite preserving the color gradient.The figure illustrates why pixel-based scores may misalign with visual quality.Source: Figure 2 · p. 2
  • StarVector uses multimodal language modeling to generate SVG code from images or text, recognizing image semantics and using SVG primitives for precise, compact graphics.The approach frames image vectorization as inverse rendering and code generation rather than pixel reconstruction.Source: §1 · p. 2
  • SVG-Bench covers 10 datasets and three tasks: image-to-SVG, text-to-SVG, and diagram generation.The paper also proposes DinoScore because pixel-based MSE can fail to reflect vector-graphic quality.Source: §1 · p. 2
  • SVG-Stack provides over 2M samples for training image-to-SVG and text-to-SVG generation.Samples pair SVGs with rendered images and text descriptions.Source: §1 · p. 2

2. Related Work

Prior vectorization methods and datasets leave gaps in semantic understanding, primitive use, task coverage, and SVG-specific evaluation. StarVector, SVG-Stack, and SVG-Bench are presented as responses to these gaps.

SVG Generation Methods

  • Traditional image-processing methods trace curves from pixels but can produce complex representations with artifacts and little semantic understanding.Deep-learning approaches improve vector modeling but often remain limited to paths and struggle to generalize [14] [47] [67].Source: §2 · pp. 2–3
  • Text-conditioned SVG approaches include iterative optimization methods and an icon-generation model restricted to paths [86] [94].The paper contrasts these approaches with training an MLLM on inverse rendering to generate SVG code using visual understanding and primitives.Source: §2 · p. 3

SVG Benchmarks and Datasets

  • Existing datasets focus on limited SVG types, while pixel-based metrics can miss fidelity and structure, and prior benchmarks cover a limited scope [12] [14] [67] [89] [94].SVG-Stack and SVG-Bench are introduced to provide broader task coverage, datasets, and SVG-tailored metrics.Source: §2 · p. 3

SVG-Stack Dataset

SVG-Stack is a large, diverse corpus for multimodal SVG generation, pairing SVG code with raster images and descriptive text. Its curation and augmentation support training across varied SVG structures and input conditions.

SVG-Stack Dataset

  • SVG-Stack contains 2.1 million training SVGs, 108k validation samples, and 5.7k test samples, each paired with a raster image and descriptive text.The samples are sourced from The Stack and undergo additional filtering for duplicates.Source: SVG-Stack Dataset · p. 3
  • The corpus includes real-world SVGs with varied syntax, templating, headers, and framework-specific implementations, alongside broad primitive support.Its permissively licensed samples are drawn from publicly available GitHub repositories.Source: SVG-Stack Dataset · p. 3
  • Synthetic captions from BLIP2 and Llava expand the corpus to 4 million paired textual captions and SVGs.The captioning augmentation supports text-to-SVG training.Source: SVG-Stack Dataset · p. 4
  • SVG-specific augmentation varies resolution, rotation, translation, scale, and color, with rasterization performed during collation.The paper reports that ablations show these techniques significantly improve performance.Source: SVG-Stack Dataset · p. 4

StarVector

StarVector treats SVG generation as conditional code prediction: visual or textual inputs condition an autoregressive language model that generates SVG sequences. Its image pathway projects encoder features into the language model’s embedding space.

4.1. Architecture

  • StarVector frames image vectorization as inverse rendering and code generation, converting images to visual tokens that precede SVG code tokens.The model is trained on SVG-Stack and learns to predict SVG code directly.Source: §4.1 · p. 4
  • A Vision Transformer encodes image patches into features, and a nonlinear adapter projects them into visual tokens aligned with the language model’s embedding space.The adapter uses fully connected layers, Swish activation, and Layer Normalization.Source: §4.1 · pp. 4–5
  • Training conditions SVG sequences on either image features or text, using next-token cross-entropy over the SVG sequence.At inference, the model autoregressively generates SVG code until the <svg-end> token.Source: §4.1 · p. 5
  • StarVector-1B uses 224×224 images, 257 visual tokens, and an 8192-token context, while StarVector-8B uses 384×384 images, 576 visual tokens, and a 16k-token context.The variants also differ in image encoder and language-model capacity.Source: §4.1 · p. 5
  • Generation can be incomplete because of stochastic decoding and limited context, and output quality is sensitive to temperature, length penalty, and logit bias.The authors generate k samples with varied parameters and rank them using DinoScore.Source: §4.1 · p. 5

SVG-Bench: Evaluation Suite for SVG

SVG-Bench unifies evaluation across Image-to-SVG, Text-to-SVG, and diagram generation, spanning varied SVG types from icons and fonts to complex diagrams. It standardizes datasets and task-specific metrics.

SVG-Bench: Evaluation Suite for SVG

  • SVG-Bench evaluates Image-to-SVG, Text-to-SVG, and diagram generation across SVGs ranging from icons and fonts to complex diagrams.Its 10 datasets cover images of varying complexity, text descriptions, and diagrams involving text, rectangles, and arrows.Source: SVG-Bench: Evaluation Suite for SVG · p. 5, §5.1 · p. 5

5.1. Tasks and Benchmarks

  • Image-to-SVG benchmarks include SVG-Fonts, SVG-Emoji, SVG-Icons, and SVG-Stack, with standardized splits and simplified path-only versions for compatibility.SVG-Diagrams is a test set extracted from SVG-Stack with textual captions.Source: §5.1 · p. 5
  • Image vectorization and diagram generation use MSE, SSIM, LPIPS, DinoScore, and SVG token length, while Text-to-SVG uses FID, FID-CLIP, and CLIP Score.DinoScore computes L2 distance between DinoV2 features; token length is the average SVG code length using the StarCoder tokenizer.Source: §5.1 · pp. 5–6
  • For image vectorization, image-processing methods excel in SSIM and MSE, while StarVector leads in DinoScore and LPIPS and produces compact outputs.StarVector performs better on SVG-Stack, SVG-Fonts, and SVG-Icons but underperforms on SVG-Emoji, attributed to limited training data.Source: Table 1 · p. 6
  • On SVG-Diagrams, StarVector outperforms LIVE in DinoScore, LPIPS, and SSIM, while LIVE ranks best in MSE.The table caption also reports that StarVector remains competitive in token length.Source: Table 2 · p. 6

6. Experiments and Results

Across SVG-Bench, StarVector leads on DinoScore on six of eight image-vectorization benchmarks and produces roughly 3k-token SVGs. It also generates diagrams with the required primitives and outperforms prior baselines on the reported Text-to-SVG metrics.

6. Experiments and Results

  • The experiments train StarVector-1B and -8B on SVG-Stack, then fine-tune for image and text tasks and evaluate models on SVG-Bench.Baselines include image-processing, deep-learning, and multimodal language models; training uses 2 epochs, with 7 days on 8 A100 GPUs for 1B and 10 days on 64 H100 GPUs for 8B.Source: §6 · pp. 6–7

6.1. Main Results

  • StarVector achieves the highest DinoScore on six of eight image-vectorization benchmarks, averaging approximately 3k tokens versus LIVE’s approximately 18k.LIVE leads on SSIM and MSE across datasets; the authors attribute StarVector’s compact outputs primarily to effective SVG primitive use.Source: §6.1 · p. 7
  • Pixel-based metrics can misrepresent SVG quality because they are sensitive to small spatial misalignments and miss non-smooth corner artifacts.The authors report that DinoScore aligns more consistently with visual quality and penalizes poorly formatted samples.Source: §6.1 · p. 7
  • In qualitative comparisons on SVG-Stack and SVG-Diagrams, StarVector-8B better preserves shape and definition, while LIVE, VTracer, and AutoTrace show artifacts on small details.Potrace produces sharper results but is monochromatic.Source: §6.1 · p. 7, Figure 4 · p. 8
  • StarVector is the only method reported as capable of diagram generation, applying rectangles, arrows, and text where other methods produce blobs and curves.DinoScore and LPIPS capture this advantage, while MSE and SSIM remain poorly aligned; human evaluations also preferred StarVector’s outputs.Source: §6.1 · p. 8
  • StarVector uses SVG primitives to represent shapes compactly, whereas VTracer relies on a large collection of paths.The paper connects primitive use with visual semantic understanding, shape composition, and direct SVG code generation.Source: §6.1 · p. 8
  • Human evaluations showed a strong preference for StarVector-8B across settings, particularly for SVG-Diagrams, despite weak correlations between MSE and human judgments.DinoScore had stronger reported correlations with human evaluation than MSE.Source: §6.1 · p. 8, Figure 5 · p. 8
  • StarVector outperforms baselines on SVG-FIGR and SVG-Stack in Text-to-SVG, though semantic accuracy often suffers due to data quality.Table 4 reports that StarVector models outperform previous baselines across all reported metrics and improve when scaled from 1B to 8B.Source: Table 4 · p. 7, §6.1 · p. 8

7. Conclusions

The paper presents StarVector as an MLLM for Image-to-SVG and Text-to-SVG, alongside SVG-Stack and SVG-Bench. It also identifies limits from context length, visual feedback, and generation speed.

7. Conclusions

  • StarVector is reported to deliver superior visual quality, precise line and shape rendering, and effective primitive use compared with baselines.The authors also describe SVG-Stack as supporting generalization across SVG types and primitives, and SVG-Bench as a unified benchmark with targeted metrics.Source: §7 · p. 8
  • StarVector is constrained by a 16k-token context for complex SVGs, relies mainly on code prediction with minimal visual feedback, and is limited in generation speed.The authors suggest future investigation of pixel-based signals.Source: §7 · p. 8

Supplementary Material

The supplementary material outlines additional dataset details, experiments, baselines, architecture and training information, SVG-Bench setup, and qualitative results.

Supplementary Material

  • Supplementary sections cover datasets, experiments, baselines, StarVector architecture and training, SVG-Bench evaluation, and qualitative examples of model strengths and limitations.Source: Supplementary Material · p. 13

8. SVG Datasets in SVG-Bench

This section describes SVG-Bench dataset construction and supporting data procedures, including simplified datasets, font and diagram data, synthetic captions, and SVG augmentation.

8. SVG Datasets in SVG-Bench

  • SVG-Bench provides accessible dataset versions with train, validation, and test splits, and includes only examples of up to 8,192 tokens.The token limit reflects StarVector’s capacity of up to 8k context tokens.Source: §8 · p. 13

8.1. Datasets with Simplified SVGs.

  • Simplified versions of the emoji, icon, font, and SVG-Stack datasets remove complex primitives, retaining paths with simple line strokes and abstracted shapes.This simplification is used because DeepSVG requires simplified SVG input.Source: §8.1 · p. 13

8.2. Creating the SVG-Fonts Dataset

  • SVG-Fonts is constructed by converting public font packages from TTF through SFD into SVG; its examples contain one character per image.The test samples contain only path elements.Source: §8.2 · p. 13

8.3. Creating SVG-Diagrams

  • SVG-Diagrams contains diagrams, graphs, workflows, and other designs with discrete elements, and targets structured designs beyond path-only approaches.Its examples include intricate structures, small text, and varied aspect ratios.Source: §8.3 · p. 13, Figure 6 · p. 13

8.4. Generating Synthetic Captions on SVG-Stack

  • SVG-Stack images are captioned with BLIP2 and Llava, and captions below a CLIP Score threshold of 30 are filtered out.Two captions are generated per image before filtering.Source: §8.4 · pp. 13–14

8.5. Data Augmentation for SVG

  • Both SVG-Emoji and SVG-Icons improve with SVG augmentations, with a substantial uplift for data-limited SVG-Emoji.The operations include rotation, color changes, and curve noise.Source: §8.5 · p. 14

9. SVG Methods and Baselines

The section describes the reproduced image- and text-to-SVG baselines evaluated in the authors’ setup, including their methods and relevant constraints.

9.1. Image-to-SVG Baselines

  • Image-to-SVG comparisons include DeepSVG, Im2Vec, LIVE, GPT-4 Vision, VTracer, Autotrace, and Potrace.The methods span deep learning, multimodal language models, and image processing.Source: §9.1 · pp. 15–16
  • LIVE progressively optimizes closed paths, but requires a fixed path count; 32 paths gave good visual results and generation took over 10 minutes per SVG.The authors describe the test-time optimization as slow for professional use.Source: §9.1 · pp. 17–18
  • DeepSVG results are reported on simplified test sets because the framework only supports simplified SVGs; it also limits outputs to eight groups and paths of at most 30 commands.The simplified representations use simple line strokes and splines.Source: §9.1 · p. 18

9.2. Text-to-SVG Generation Baselines

  • Text-to-SVG baselines were selected for reproducible public methods or code and include CodeLlama, zero-shot GPT-4, and IconShop.IconShop was trained on its original dataset and splits using a pre-trained checkpoint.Source: §9.2 · p. 19

10. Additional Experiments and Results

These experiments show StarVector’s results across image-, text-, and diagram-to-SVG tasks, alongside limitations tied to context length, image processing, and training-data coverage.

10.0.1. Image-to-SVG Results

  • Simplified image datasets are converted nearly perfectly, while emoji outputs capture shape, color, and semantics but can misplace objects; icons may be incomplete when code tokens run out.The authors attribute emoji positioning errors to fewer than 10,000 training examples.Source: §10.0.1 · p. 19
  • The authors report that high-fidelity SVG generation is achievable with 1B parameters, while LLM-based approaches offer greater generalization and scalability at the cost of more parameters.They note that GPT-4V lacks specific training for SVG generation.Source: §10.0.1 · p. 26
  • Benchmark comparisons are restricted to 8k context; complex structures challenge fixed-size 224- or 384-pixel image inputs, and uncommon styles expose gaps in training-data diversity.The authors suggest broader training data, including niche applications and artistic domains, as one direction for improving generalization.Source: §10.0.1 · p. 26

10.1. Ablation Studies

  • CLIP consistently outperforms VQGAN and ConvNext across tested datasets and reconstruction metrics, with the latter encoders tending to lose local detail.Training the CLIP image encoder on SVG-Stack improves results on SVG-Icons and SVG-Fonts.Source: §10.1 · p. 26, Table 9 · p. 28, Figure 21 · p. 28
  • SVG-Stack pretraining improves metrics on smaller datasets and is described as vital for SVG-Emoji, where training without it overfits.The augmentation and pretraining ablations use StarVector-1B on smaller datasets susceptible to overfitting.Source: §10.1 · p. 27, Table 8 · p. 28
  • Beam search with size 5 achieves the best outcomes but uses more memory and runtime; nucleus sampling with top-p 0.9 and temperature 0.5 gives the best overall performance.The sampling ablation also tests greedy decoding and other temperature settings.Source: §10.1 · p. 27, Table 7 · p. 27

10.2. Text-to-SVG Results

  • Text-conditioned outputs generally capture colors, shapes, and semantics, but may omit requested details or produce incoherent shapes.The authors associate some errors with model scale and context length, and suspect imprecise or ungrounded textual descriptions also contribute.Source: §10.2 · p. 27

10.3. Results on SVG-Diagrams

  • StarVector-8B is the only compared approach reported to produce plausible diagram results; DinoScore aligns with that assessment, while MSE favors baselines that lose diagram semantics and structure.The comparisons include LIVE, VTracer, Potrace, and AutoTrace.Source: §10.3 · p. 28

10.4. Analysis of SVG Primitives

StarVector uses the broader SVG language to produce compact, semantically aware graphics, and human evaluators prefer its outputs despite pixel-metric advantages for some baselines. Comparisons highlight trade-offs among semantic richness, reconstruction scores, and speed.

10.4. Analysis of SVG Primitives

  • StarVector identifies basic shapes and uses the full SVG syntax, producing compact, interpretable outputs unlike path-restricted baselines.Its generated SVG tag distribution is reported as similar to SVG-Stack’s ground truth.Source: §10.4 · p. 29, Table 10 · p. 36, Figure 30 · p. 37

10.5. Human Evaluation

  • Human evaluators strongly prefer StarVector-8B across settings, especially for SVG-Diagrams, despite pixel-based metrics favoring some baselines.DinoScore correlates more strongly with human judgments than MSE in the reported analyses.Source: §10.5 · p. 29

10.6. Comparing StarVector with Baselines

  • DeepSVG is effective on simplified datasets but limited there, while Im2Vec shows poor generalization on the evaluated SVG-Bench datasets.Im2Vec’s evaluation results are described as poor enough that qualitative samples are not presented.Source: §10.6 · p. 29
  • LIVE achieves the best MSE, LPIPS, and SSIM results, but its path-only SVG files contain 19k tokens versus StarVector’s compact 3k-token outputs using varied SVG primitives.DiffVG also produces approximately 19k-token files and requires 30 seconds per sample, substantially faster than LIVE.Source: §10.6 · pp. 37–38
  • VTracer, Potrace, and AutoTrace generate quickly, but struggle with diagrams and fine details and cannot encode text semantically.VTracer and AutoTrace take under a second, while Potrace typically takes around 10 seconds; StarVector requires over a minute.Source: §10.6 · p. 38
  • IconShop performs well on SVG-FIGR but is restricted to path modeling; StarVector outperforms it across reported Text-to-SVG metrics.The comparison includes FID, FID CLIP, and CLIP Score.Source: §10.6 · p. 38
  • StarVector is the only deep learning-based Image-to-SVG model reported to achieve results comparable to tracing methods, and it outperforms other evaluated LLMs.The authors also report strong Text-to-SVG results against IconShop on FIGR-SVG.Source: §10.6 · p. 38

10.7. Human Evaluation

A blinded, randomized human evaluation found a strong preference for StarVector-8B over baselines, particularly on SVG-Diagrams. DinoScore aligned more closely with human judgments than MSE did.

10.7. Human Evaluation

  • Human evaluators strongly preferred StarVector-8B across settings, especially on SVG-Diagrams, favoring its sharp shapes and effective use of primitives.Participants were screened for conflicts of interest, and outputs were anonymized and randomized.Source: §10.7 · p. 39
  • DinoScore correlated more strongly with human evaluations than MSE, including a 0.7577 correlation between DinoScore differences and human-evaluation scores.Reported MSE correlations were 0.0596 and -0.1002; DinoScore correlations were -0.6193 and 0.6214.Source: §10.7 · p. 39

11. StarVector Method

StarVector combines a pretrained language model with an image encoder to generate SVG code from images, then fine-tunes for text-prompted SVG generation. Its training and sampling procedures use token prediction and stochastic decoding, with post-processing for incomplete outputs.

11.1.1. Large Language Model

  • StarVector uses StarCoder-based language-model variants chosen for long SVG contexts, decoding speed, and pretrained coding capability.StarVector-1B uses StarCoderBase-1B, while StarVector-8B uses StarCoder2-7B.Source: §11.1.1 · p. 39

11.1.2. Image Encoder

  • CLIP performed best among tested image encoders, which the authors attribute in part to its 257 visual tokens.The image encoder is used only for Image-to-SVG; the 224×224 resolution can lose detail in diagrams with extreme aspect ratios, including rendered text.Source: §11.1.2 · pp. 39–40
  • Natural-image SVG generation is outside this project’s scope, and the authors identify adapting the approach to natural images as future work.The image encoder comparisons also report that VQGAN and ConvNext often fail to capture image shape and path trajectory.Source: §11.1.2 · p. 40

11.2. Training

  • Training has two stages: image-to-SVG next-token learning on SVG-Stack, followed by text-to-SVG fine-tuning without the image encoder.The first stage uses cross-entropy next-token prediction and does not require image-domain pixel-loss supervision.Source: §11.2 · p. 41

11.3. Generation

  • Generation uses nucleus sampling with top-p=0.9 and temperature 0.5–0.9 as the recommended setting; beam search increases inference time when combined with nucleus sampling.The authors report that 85% of generated SVGs compile within context length; post-processing makes all outputs valid and compilable, though image parts may be lost.Source: §11.3 · p. 41
  • Sampling five outputs and selecting by DinoScore raised the reported score by 0.12; 32% of sampled test cases were more accurate, at increased generation time.The comparison used StarVector-8B on SVG-Stack, with k=1 versus k=5.Source: §11.3 · p. 42

Related papers

Loading 2312.11556v4…