PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniSVG: A Unified Scalable Vector Graphics Generation Model

Yiying Yang, Wei Cheng, Sijin Chen, Xianfang Zeng, Fukun Yin, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, Yu-Gang Jiang

arXiv:2504.06263v3cs.CV

TL;DR

Existing SVG generation methods either produce costly, unstructured outputs or handle only simplified content. OmniSVG uses pre-trained VLMs and discrete SVG tokenization for end-to-end multimodal generation, and experiments report superiority over prior methods. MMSVG-2M and MMSVG-Bench extend the resource and evaluation infrastructure for complex SVG synthesis.

  • Problem

    Existing methods are computationally costly and unstructured or limited to simplified SVG content, while complex SVG datasets and standardized evaluations remain scarce.

  • Method

    OmniSVG combines pre-trained VLMs with discrete tokenization of SVG commands and coordinates to generate complex multimodal SVGs end to end.

  • Results

    OmniSVG surpasses prior SVG generation methods quantitatively and qualitatively across various conditional generation tasks.

  • Takeaways & Limitations

    MMSVG-2M and MMSVG-Bench provide a two-million-asset multimodal resource and standardized evaluation protocol for future SVG generation research and workflows.

  • Takeaways & Limitations

    Complex samples require tens of thousands of generated tokens and considerable generation time, and OmniSVG fails on natural-image prompts.

Abstract

from arXiv · show

Scalable Vector Graphics (SVG) is an important image format widely adopted in graphic design because of their resolution independence and editability. The study of generating high-quality SVG has continuously drawn attention from both designers and researchers in the AIGC community. However, existing methods either produces unstructured outputs with huge computational cost or is limited to generating monochrome icons of over-simplified structures. To produce high-quality and complex SVG, we propose OmniSVG, a unified framework that leverages pre-trained Vision-Language Models (VLMs) for end-to-end multimodal SVG generation. By parameterizing SVG commands and coordinates into discrete tokens, OmniSVG decouples structural logic from low-level geometry for efficient training while maintaining the expressiveness of complex SVG structure. To further advance the development of SVG synthesis, we introduce MMSVG-2M, a multimodal dataset with two million richly annotated SVG assets, along with a standardized evaluation protocol for conditional SVG generation tasks. Extensive experiments show that OmniSVG outperforms existing methods and demonstrates its potential for integration into professional SVG design workflows.

1 Introduction

SVG generation remains difficult because optimization methods are computationally expensive and unstructured, while autoregressive methods struggle with complex content. OmniSVG addresses these limitations with VLM-based multimodal generation, supported by a two-million-asset dataset and standardized benchmark.

  • Optimization-based SVG methods incur substantial computational overhead and can produce redundant anchor points that harm editability.
  • Autoregressive SVG methods scale better through end-to-end learning but remain limited by context length and scarce complex SVG data.
  • OmniSVG parameterizes SVG coordinates and commands as discrete tokens, separating structural logic from geometry and supporting SVGs exceeding 30k tokens.
  • OmniSVG uses native VLMs for end-to-end multimodal generation across Text-to-SVG, Image-to-SVG, and Character-Reference SVG tasks.
  • MMSVG-2M provides two million richly annotated SVG assets, while MMSVG-Bench standardizes evaluation for Text-to-SVG and Image-to-SVG generation.
  • Experiments report that OmniSVG surpasses prior SVG generation methods both quantitatively and qualitatively.

2 Related Works

Prior SVG generation spans neural sequence models, LLM-based code synthesis, and image vectorization, but complex structures remain difficult to represent and edit. OmniSVG combines native VLMs with SVG command parameterization to target multimodal generation of complex SVGs.

  • Earlier SVG generators used RNNs, VAEs, and Transformers to compress SVG commands into latent representations, with DeepSVG struggling with geometric consistency.
  • LLM-based SVG generation through XML or code synthesis is constrained by context length when complex SVGs exceed 10k tokens.
  • OmniSVG combines pre-trained VLMs with SVG command parameterization to follow multimodal instructions and generate vivid, complex SVGs.
  • Diffusion-based image vectorization can suffer from over-smoothing, color over-saturation, tangled paths, and weak editability or hierarchical structure.
  • Existing SVG datasets emphasize simplified paths or monochrome icons, with samples generally below 8.2k tokens and limited illustrative coverage.

3 OmniSVG Dataset

MMSVG-2M is a large-scale, multimodal SVG dataset spanning icons, illustrations, and anime characters, paired with MMSVG-Bench for conditional generation evaluation. The data are standardized through deduplication, captioning, and atomic command simplification, while the benchmark uses task-specific metrics.

  • MMSVG-2M: MMSVG-2M contains two million SVG samples covering website icons, illustrations, graphic designs, anime characters, and related categories.
  • MMSVG-2M: The dataset comprises 1.1 million icons, 0.5 million illustrations, and 0.4 million anime characters collected or created from multiple sources.
  • MMSVG-2M: Curation removes duplicates, fits SVGs within a 200 × 200 viewbox, and uses BLIP-2 to generate captions.
  • MMSVG-2M: SVG commands are simplified into five atomic path commands—M, L, C, A, and Z—to reduce ambiguity and standardize representation.
  • SVG representation: Table 1 documents the draw commands, their arguments, and visualizations, with each command’s start position inherited from the preceding command’s endpoint.
  • MMSVG-Bench: MMSVG-Bench evaluates Text-to-SVG using FID, Aesthetic, CLIP, and HPS metrics, and Image-to-SVG using DinoScore, SSIM, LPIPS, and MSE.
  • MMSVG-Bench: Character-Reference SVG Generation assesses whether generated SVGs preserve an input character’s profile using GPT-4o scores from 1 to 10.

4 OmniSVG

OmniSVG represents complex, colored SVGs as discrete token sequences for end-to-end multimodal generation with a pretrained VLM. Its tokenizer preserves path structure while compacting coordinates, and experiments report superior performance against current SVG-generation baselines.

  • Architecture: OmniSVG parameterizes atomic SVG path commands into sequences before feeding them to a pretrained VLM with multimodal instructions.The approach supports end-to-end training for multimodal SVG generation.
  • SVG Representation: Each SVG script is represented as multiple paths, with each path containing an ordered set of commands.The representation separates path-level organization from the individual commands within each path.
  • SVG Representation: Commands encode both a command-type identifier and its corresponding location argument, using M, L, C, A, and Z for basic path operations.This parameterization explicitly represents command structure alongside geometry.
  • Color Representation: Colored SVGs add a special F command type for hex-value tokens that control the Fill attribute.The colored representation expands the command types to M, L, C, A, Z, and F.
  • Tokenization: The tokenizer flattens paths into one ordered sequence, adds SOP and EOS boundaries, and merges each 2D coordinate pair into one token using x × w + y.The resulting sequence is embedded in the pretrained VLM’s representation space through a learnable embedding layer.
  • Training: OmniSVG predicts SVG suffix tokens conditioned on multimodal instruction prefix tokens using the standard next-token prediction objective.The objective is described as P(xs,i | xs,<i, xc).
  • Evaluation: OmniSVG demonstrates superior performance compared with current state-of-the-art text-to-SVG and image-to-SVG baseline methods.Table 2 reports quantitative evaluations across both conditional generation tasks.

5 Experiments

Experiments compare OmniSVG with prior methods across text-to-SVG, image-to-SVG, character-reference, parameterization, model-size, architecture, and user-study settings. Results indicate strong visual quality, efficiency, multimodal coverage, and benefits from full parameterization and larger VLMs.

  • Quantitative Comparisons: OmniSVG is compared with language-based, vectorization, and multimodal baselines on text-to-SVG and image-to-SVG tasks.The evaluation also reports generated SVG token length.
  • Quantitative Comparisons: OmniSVG achieves superior FID scores and competitive CLIP score, aesthetic quality, and HPS on text-to-SVG generation.
  • Qualitative Evaluations: OmniSVG outperforms text-to-SVG baselines across prompts with high-fidelity, colorful, geometrically accurate, and semantically consistent outputs.IconShop produces monochrome SVGs, while Chat2SVG generates less detailed and semantically consistent results in its first stage.
  • Qualitative Evaluations: For image-to-SVG, OmniSVG outperforms the baselines in quality and efficiency and converts icons, illustrations, and character images into editable SVGs.Traditional optimization methods struggle with complex images, while several autoregressive baselines have limited generalization or output complexity.
  • Qualitative Evaluations: Character-reference training on paired character images and SVGs enables OmniSVG to generate character SVGs from image references.
  • Ablation Studies: Parameterizing both coordinates and colors yields better results under all metrics with the shortest token length.Qualitative results show the advantage becomes especially pronounced as SVG complexity increases, while non-parameterization fails on complex images.
  • Ablation Studies: Larger OmniSVG base models generate higher-quality SVG samples, and Qwen2.5-VL outperforms alternative VLM architectures under all evaluation metrics.
  • User Study: In a 15-participant user study, OmniSVG is widely preferred and receives higher vividness and semantic-alignment scores than baseline models.Participants evaluated outputs from 150 text descriptions and 150 image prompts.

6 Conclusions

OmniSVG is presented as a unified VLM-based framework for generating high-quality, editable SVGs across conditional multimodal tasks. The conclusion highlights strong comparative results and professional-workflow potential, while acknowledging inference cost and failure on natural-image prompts.

  • Conclusions: OmniSVG uses pre-trained VLMs and discrete command-coordinate tokens to decouple SVG structural logic from geometry.
  • Conclusions: OmniSVG outperforms existing approaches in quality and efficiency while producing high-quality, editable SVGs across design domains.
  • Conclusions: Extensive experiments show that OmniSVG surpasses prior methods across various conditional generation tasks, supporting integration into professional SVG workflows.
  • Limitations and Future Work: During inference, complex samples require tens of thousands of tokens, causing considerable generation time.
  • Limitations and Future Work: OmniSVG is bounded by vector-style image prompts and fails on natural images.

A.1 Samples of MMSVG-2M Dataset

MMSVG-2M contains two million SVG assets spanning icons, illustrations, and anime characters, with rasterized images, generated captions, and character-reference pairs. Its character subset combines internet-sourced and pipeline-generated data for multimodal prompting.

  • Dataset Composition: MMSVG-2M includes Icon, Illustration, and Character subsets, with character-reference data built from natural character images paired with SVGs.
  • Dataset Composition: MMSVG-2M contains 1.1 million icons, 0.5 million illustrations, and 0.4 million anime characters.
  • Dataset Construction: The dataset comprises two million SVG samples with corresponding rasterized images and BLIP-2-generated captions for instruction fine-tuning.CairoSVG performs rasterization, and completely white outputs are removed.
  • Annotation: BLIP-2 captions are filtered by dropping samples with CLIP scores below 30 to reduce hallucinations.
  • Dataset Construction: Part of the Character subset is generated with a FLUX-based vector-style pipeline, while another part is downloaded from the Internet.FLUX-Redux injects image features through a SigLIP encoder for image-based conditioning.
  • Scaling Study: Scaling OmniSVG from 4B to 8B parameters lowers validation perplexity when both models train from scratch on 250 billion tokens.

B.2 Implementation Details

Implementation uses memory-efficient bfloat16 training, ZeRO-2, AdamW with a decaying learning rate, pretrained Qwen2.5-VL weights, and top-k/top-p sampling for diversity.

  • Training: Models are trained in bfloat16 with ZeRO-2 and AdamW, using a learning rate decaying from 3×10−4 to 3×10−6 and weight decay 0.1.
  • Initialization: Training initializes from pretrained Qwen2.5-VL weights while SVG embeddings are initialized from scratch.
  • Generation: SVG generation uses top-k sampling with k = 50 and top-p sampling with p = 0.95 to promote diversity.

C Additional Results

The additional results compare OmniSVG with established image-vectorization methods and visualize dataset labels, prompting choices, and scaling behavior.

  • Comparative evaluation: Text-to-SVG and image-to-SVG evaluations compare OmniSVG against language-based, image-vectorization, multimodal, and raster-to-vector baselines.Compared methods include VectorFusion, SVGDreamer, Chat2SVG, IconShop, LIVE, DiffVG, StarVector, Vtracer, and GPT-4o.
  • Dataset analysis: MMSVG-2M label frequencies are represented by word-cloud label sizes, with larger labels indicating more frequent occurrences.
  • Dataset analysis: Image-prompting pairs for MMSVG-2M Character are generated with FLUX-Redux and SVG vectorization tools, using downsampling scales 2 and 3 to trade off similarity and SVG complexity.
  • Scaling analysis: Training and validation perplexity results are reported for OmniSVG models trained from scratch on 250 billion tokens, with performance increasing as model size grows.

D.1 Text-to-SVG Task

The text-to-SVG discussion surveys optimization, diffusion, language-model, and multimodal approaches, emphasizing their differing mechanisms and structural limitations.

  • Text-to-SVG methods: SVGDreamer separates foreground and background through semantic-driven image vectorization, while attention-based primitive control improves manipulation of individual elements.
  • Text-to-SVG methods: VectorFusion distills semantic knowledge from a text-conditioned diffusion model into SVGs by optimizing a differentiable vector graphics rasterizer.
  • Text-to-SVG methods: Chat2SVG combines LLM-generated geometric SVG templates with image-diffusion-guided optimization to refine paths and point coordinates.
  • Text-to-SVG methods: IconShop autoregressively models SVG path-command sequences with a transformer and reports strong results in simplified icon scenarios.
  • Text-to-SVG methods: LLM4SVG uses learnable semantic tokens for structured SVG encoding but has a 2048-token maximum length that limits highly complex SVG generation.
  • Related approaches: LIVE, DiffVG, StarVector, and Vtracer represent alternative SVG-generation or vectorization pipelines using progressive path optimization, differentiable rasterization, visual-language modeling, or hierarchical clustering.LIVE recursively optimizes paths; DiffVG computes gradients through rasterization; StarVector generates SVG code from visual inputs; Vtracer simplifies pixels into polygons and Bézier curves.
Loading 2504.06263v3…