PaperlayerView exact revision on arXiv

Source-linked AI summary

SVGDreamer: Text Guided SVG Generation with Diffusion Model

Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, Qian Yu

arXiv:2312.16476v7cs.CVcs.AI

TL;DR

Existing text-to-SVG methods have limited editability, visual quality, and diversity. SVGDreamer combines semantic-driven vectorization with particle-based score distillation, and experiments report improvements across those dimensions.

  • Problem

    Existing text-to-SVG methods have limited editability, while SDS-based approaches can produce over-smoothed shapes, over-saturated colors, limited diversity, and slow convergence.

    Source: §1 · p. 1

  • Method

    SVGDreamer uses SIVE to decompose foreground objects and background elements, and VPSD to model SVG control points and colors as distributions for refinement.

    Source: §1 · pp. 1–2

  • Results

    SVGDreamer consistently outperforms existing baselines in editability, visual quality, and diversity across diverse text prompts and vector styles.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pipeline can create reusable SVG assets for icons, decorative elements, and concept-style illustrations.

    Source: §4.4 · p. 9

    Editability depends on the T2I model's decomposition capability, and SIVE does not yet adaptively determine each object's control-point count.

    Source: §5 · p. 9

Abstract

from arXiv · show

Text-guided scalable vector graphics (SVG) synthesis has broad applications in icon and sketch generation. However, existing text-to-SVG methods often suffer from limited editability, suboptimal visual quality, and low sample diversity. To address these challenges, we propose \textbf{SVGDreamer}, a novel framework for text-guided vector graphics synthesis. Our method introduces a \textbf{semantic-driven image vectorization (SIVE)} process, which decomposes the generation procedure into foreground objects and background elements, thereby improving structural controllability and editability. In particular, SIVE incorporates attention-based primitive control and an attention-mask loss to facilitate fine-grained manipulation of individual vector elements. To further improve generation quality and diversity, we propose \textbf{Vectorized Particle-based Score Distillation (VPSD)}, which models SVGs as distributions over control points and colors. Compared with existing text-to-SVG optimization methods, VPSD alleviates over-smoothed shapes, over-saturated colors, limited diversity, and slow convergence. Moreover, VPSD leverages a reward model to reweight vector particles, leading to better visual aesthetics and faster convergence. Extensive experiments demonstrate that SVGDreamer consistently outperforms existing baselines in editability, visual quality, and diversity. Project page: https://ximinng.github.io/SVGDreamer-project/

1. Introduction

Existing text-to-SVG methods offer limited object-level editability and can produce over-smoothed, over-saturated, low-diversity results. SVGDreamer combines semantic decomposition with distribution-based refinement, and experiments report gains in editability, visual quality, and diversity.

1. Introduction

  • Existing methods jointly synthesize primitives without disentangling foreground from background, making semantic components entangled and object-level editing difficult.SDS-based optimization is also described as mode-seeking, potentially suppressing fine details or missing prompted objects.Source: §1 · p. 1
  • SIVE separates foreground objects from background elements and uses attention-based primitive control and an attention-mask loss for hierarchical, object-level editing.Cross-attention maps queried by text tokens initialize control points for individual objects and background regions.Source: §1 · pp. 1–2
  • VPSD represents SVGs as distributions over control points and colors, addressing SDS-related over-smoothing, over-saturation, limited diversity, and slow convergence.A LoRA-based network aligned with a pretrained diffusion model estimates the distributions; reward feedback further improves perceptual quality and aesthetic appeal.Source: §1 · pp. 1–2
  • SVGDreamer consistently outperforms existing baselines in editability, visual quality, and diversity across diverse text prompts and vector styles.The paper also reports strong generalization across different vector-graphics categories.Source: §1 · p. 2

2. Related Work

Vector-graphics generation has progressed from data-driven sequence prediction and CLIP-guided optimization to methods using pretrained text-to-image diffusion models. Related work also motivates distribution-based score distillation as a response to limitations of deterministic SDS.

2.1. Vector Graphics Generation

  • Data-driven sequence models predict vector commands or token sequences [1, 5, 16, 23, 41, 42, 44], but depend on large vector datasets and generalize less well to open-domain or structurally complex graphics.Optimization-based approaches instead match rasterized outputs to target images or text using differentiable rendering, including DiffVG.Source: §2.1 · p. 2
  • Text-guided methods progressed from CLIP supervision, including CLIPDraw, CLIP-CLOP [18], and CLIPasso, to diffusion-based VectorFusion and DiffSketcher.Pretrained text-to-image models provide stronger visual priors for vector graphics; this work also targets editability and diversity.Source: §2.1 · p. 2, §2.2 · p. 2

2.3. Score Distillation Sampling

  • SDS-based vector methods inherit over-smoothed shapes and limited diversity; VPSD instead models control points and colors as distributions and adds reward-guided refinement.SDS is characterized as deterministic and mode-seeking; Figure 1 describes style control through vector primitives rather than a fixed prompt suffix.Source: Figure 1 · p. 3, §2.3 · p. 4

3. Methodology

SVGDreamer combines semantic-driven vectorization with particle-based score distillation to generate prompt-aligned SVGs. It separates semantic regions before refining vector-parameter distributions and supports style control through primitive configurations.

3. Methodology

  • SVGDreamer represents SVGs as paths with color attributes and optimizes their parameters through a differentiable renderer guided by a pretrained text-to-image diffusion model.Its two components are SIVE and VPSD.Source: §3 · p. 4

3.1. SIVE: Semantic-driven Image Vectorization

  • SIVE uses diffusion cross-attention maps and an attention-mask loss to separate foreground objects from background elements and hierarchically optimize editable vector objects.The method aims to improve object separation, controllability, and editability.Source: §3 · p. 4, §3.1 · p. 4

3.1.1 Primitive Initialization

  • Object-token cross-attention maps assign paths to semantic objects and initialize control points; remaining points are sampled within a radius of 0.05 times the image size.The remaining region is treated as background, and control-point sampling uses normalized attention maps as spatial distributions.Source: §3.1.1 · pp. 4–5

3.2. Vectorized Particle-based Score Distillation

  • VPSD represents SVG parameters as a conditional distribution over control points and colors, approximated by particles whose rendered images train an auxiliary score estimator.The VPSD gradient contrasts the pretrained diffusion model's target score with the estimator's current score; the estimator uses a LoRA adaptation of the pretrained model.Source: §3.2 · p. 6
  • A pretrained reward model scores estimator-generated samples, and reward-weighted feedback refines the estimator to improve aesthetic quality and accelerate VPSD convergence.The method uses DDIM to generate k samples early in optimization.Source: §3.2 · p. 6

3.3. Vector Representation Primitives

  • Users can control vector style through text or by selecting primitive sets, with six styles represented by different primitive types and parameterizations.The styles are iconography, sketch, pixel art, low-poly, painting, and ink-and-wash painting.Source: §3.3 · p. 6

4. Experiments

Across qualitative and quantitative evaluations, SVGDreamer produces more editable, visually coherent, diverse, and prompt-aligned SVGs than the compared methods. SIVE improves semantic organization, while VPSD addresses shortcomings of deterministic SDS optimization; the framework is also applied to reusable design assets.

4.1. Qualitative Evaluation

  • Compared with CLIPDraw and SDS-based methods, SVGDreamer produces clearer structure, sharper local details, more recognizable object geometry, and more stable color layouts.The qualitative comparison also reports higher aesthetic scores for final SVGDreamer outputs than for SIVE alone.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2. Quantitative Evaluation

  • The quantitative benchmark evaluates six vector styles with 10 text prompts per style and 50 generated SVGs per prompt, measuring text-guided synthesis quality and generalization across distinct primitive spaces and rendering styles.Reported metrics are PSNR, CLIPScore, BLIPScore, Aesthetic score, and Human Preference Score (HPS).Source: §4.2 · p. 7
  • SVGDreamer achieves the best overall performance across semantic alignment, visual quality, and diversity, and outperforms competing methods on FID and PSNR.Reward feedback further improves aesthetic score and HPS over the from-scratch version.Source: §4.2 · pp. 7–8

4.3.1 SIVE vs. LIVE [17]

  • Against LIVE, SIVE produces clearer foreground-background separation and more concise, interpretable path organization, supporting object-level editing.LIVE can entangle paths across semantic subjects, particularly in complex graphics.Source: §4.3.1 · p. 8, Figure 5 · p. 8

4.3.2 VPSD vs. LSDS [11, 12] and ASDS [46]

  • VPSD outperforms SDS-based variants [11, 12, 46] on FID and PSNR, with qualitative results showing sharper structures, more complete objects, and fewer missing parts.These comparisons support modeling vector graphics as parameter distributions rather than a single deterministic solution.Source: §4.3.2 · p. 9

4.4. Applications of SVGDreamer

  • SVGDreamer generates reusable vector assets for poster, logo, icon, decorative-element, and concept-illustration design.Designers can recombine generated elements with glyphs and layout components.Source: §4.4 · p. 9

5. Conclusion

SVGDreamer combines SIVE and VPSD to generate SVGs with improved editability, visual quality, and diversity. Its editability is constrained by the T2I model’s semantic decomposition, and SIVE does not yet allocate control points adaptively.

5. Conclusion

  • SVGDreamer combines SIVE and VPSD to improve editability, visual quality, and diversity in text-guided vector graphics synthesis.The paper presents the framework as a step toward broader applicability in graphic design.Source: §5 · p. 9
  • Inaccurate or incomplete semantic separation by the underlying T2I model may limit object-level controllability in generated SVGs.The authors expect advances in T2I diffusion models to improve the framework’s semantic decomposition.Source: §5 · p. 9
  • SIVE does not yet determine each object’s number of control points adaptively; automatic object-level primitive allocation remains future work.The authors identify this as an important direction for future work.Source: §5 · p. 9

Overview

The supplementary material collects additional results, application examples, implementation details, and analyses of SVGDreamer. It also includes ablations, raster-synthesis examples, and pseudo-code.

Overview

  • Section A adds qualitative results demonstrating editability, visual quality, and diversity, while Section B presents poster and icon design applications.Source: Overview · p. 12
  • Sections C and D provide implementation details and explain how SIVE prompts identify semantic objects.Source: Overview · p. 12
  • Section E ablates CFG weights, ReFL, particle count, and path count; Sections F and G show VPSD raster-synthesis examples and provide SVGDreamer pseudo-code.The overview states that the code is publicly available.Source: Overview · p. 12

A. Additional Qualitative Results

The supplementary results show that SVGDreamer can decompose and recombine SVG assets, vary styles through text and primitive choices, and support poster design. Its poster pipeline generates graphics separately from typography and layout.

A.1. Editability.

  • Generated SVGs can be split into semantically meaningful foreground and background assets, then recombined across samples into new compositions.This supports reusing or rearranging components without redrawing them from scratch.Source: §A.1 · p. 12

A.2. Visual Quality and Diversity.

  • Different primitive settings produce distinct styles, and text prompts can specify styles such as Van Gogh, pixel art, and ink wash without reference style images.Closed filled primitives suit compact color regions, while open curves and sparse strokes produce sketch-like or ink-wash appearances.Source: §A.2 · p. 12

B. Applications of SVGDreamer

B.1. Poster Design.

  • The poster pipeline generates graphical content with SVGDreamer, creates vector glyphs with FreeType, and combines both on an SVG canvas.Separating graphics from typography avoids raster T2I text-rendering failures and leaves both components editable.Source: §B.1 · p. 13
  • Compared with raster T2I baselines, SVGDreamer posters show clearer typography and more stable composition, while baseline outputs exhibit text errors or layout artifacts.GlyphControl may omit letters or interfere with imagery, while TextDiffuser control masks can reduce coherence between text and background.Source: §B.1 · p. 13

B.2. Icon Design.

SVGDreamer’s generated graphics can serve as reusable assets in icon layouts, which are completed with manually designed shapes and text. The examples include badge, circular-logo, and product-branding formats.

B.2. Icon Design.

  • Complete icons combine SVGDreamer’s core graphic content with manually designed geometric containers and vectorized text.The construction process defines polygonal or circular layouts in the SVG and appends vectorized text paths.Source: §B.2 · pp. 13–14
  • SVGDreamer’s generated graphics remain compatible with icon structures, allowing scene- or object-centric content to be composed with borders, labels, and brand text.The examples include embedding temple scenes in emblems and placing an astronaut or cup alongside circular borders or product labels.Source: §B.2 · p. 13
  • The resulting assets can be aligned with manual layouts and typography while retaining vector editability, scalability, and compositional flexibility.The glyphs shown in the icon examples are manually added.Source: §B.2 · pp. 14–15, Figure 9 · p. 15
  • Poster examples show clearer text and more editable layouts than raster text-to-image baselines.The comparison includes the input prompts and glyphs.Source: Figure 8 · p. 15

C. Implementation Details

Implementation uses six vector particles and CFG 7.5 in most experiments, while SIVE identifies semantic objects through cross-attention responses rather than a dedicated token-selection rule. Additional ablations examine CFG effects on diversity and stability.

C. Implementation Details

  • Most experiments use six particles and CFG 7.5; this setting balances prompt fidelity and diversity, while optimization requires at least 31 GB of GPU memory on an NVIDIA V100 for six parallel SVGs.SVG parameters use Adam; LoRA parameters use AdamW, and path control points follow a 700-iteration learning-rate schedule.Source: C. Implementation Details · p. 15

D. Object Identification in SIVE Prompts

  • SIVE handles prompts whose objects do not map one-to-one to tokens by relying on overlapping cross-attention maps; users can inspect those maps to select tokens for more precise control.The experiments omit dedicated token selection because semantically related tokens usually yield comparable vectorization results for prompts with clear, concentrated object attention.Source: D. Object Identification in SIVE Prompts · p. 16, Figure 10 · p. 16

E. Additional Ablation Studies

E.1. Ablation on CFG Weights

The ablations examine CFG, reward feedback, particle count, and path budget. ReFL improves efficiency and generated-SVG aesthetics, while CFG and representation budgets involve trade-offs between diversity, stability, detail, and computational cost.

E.1. Ablation on CFG Weights

  • Smaller CFG values generally increase diversity, but values that are too low, such as CFG=2, weaken semantic guidance and can destabilize optimization; the implementation uses CFG 7.5.Compared with SDS-based methods, VPSD has a more favorable guidance-strength/diversity trade-off and can produce more diverse results by reducing CFG.Source: §E.1 · p. 17

E.2. Ablation on ReFL

  • ReFL reduces optimization steps from 500 to 300, nearly halves runtime, and improves SVG aesthetic scores and qualitative structure, semantics, and details.The runtime reduction is reported under both experimental settings.Source: §E.2 · p. 17, Table 2 · p. 17, Figure 12 · p. 17

E.3. Ablation on the Number of Vector Particles

  • Increasing particle count slightly raises generation diversity while visual quality remains relatively stable; six particles are used as a compromise because larger counts add computational cost for limited diversity gains.Source: §E.3 · p. 18, Figure 13 · p. 18

E.4. Ablation on the Number of Paths

  • Increasing the path budget from 128 through 256 to 512 and 768 improves structural completeness, local detail, and semantic fidelity, with limited paths forcing a trade-off between detail and coverage.At higher budgets, sharper edges and finer elements such as reflections, foliage, and architectural contours are better preserved.Source: §E.4 · pp. 18–19, Figure 14 · p. 19

F. VPSD for 2D Image Synthesis

VPSD also applies to 2D image synthesis: the reported comparison shows more complete structures, more stable compositions, and better visual quality than VSD. The accompanying algorithms describe VPSD and its combination with SIVE.

F. VPSD for 2D Image Synthesis

  • VPSD produces 2D images with more complete structures, more stable layouts, and better overall visual quality than VSD.VSD results are described as having incomplete structures, unstable compositions, or inconsistent details.Source: Figure 15 · p. 19, F. VPSD for 2D Image Synthesis · p. 19

G. Algorithm for VPSD

  • VPSD iteratively samples an SVG parameter set, renders it to an image, updates the selected set, and updates the LoRA estimator from generated samples before returning the particle set.The algorithm initializes a frozen pretrained diffusion model, a LoRA estimator, and a pretrained reward model.Source: G. Algorithm for VPSD · pp. 19–20
  • The full SIVE-plus-VPSD pipeline extracts cross-attention maps from an initial diffusion sample, builds foreground and background masks, optimizes SVG parameters with SIVE, then refines them with VPSD.Source: G. Algorithm for VPSD · pp. 19–20

Related papers

Loading 2312.16476v7…