PaperlayerView exact revision on arXiv

Source-linked AI summary

VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models

Ajay Jain, Amber Xie, Pieter Abbeel

arXiv:2211.11319v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Designing scalable, compact SVG graphics is difficult, while diffusion models are trained primarily on raster images and lack vector decompositions. VectorFusion optimizes a differentiable vector renderer with score distillation from a pretrained diffusion model, optionally initialized from an image sample, and produces diverse vector styles with reported gains over prior approaches. Its scope remains tied to Stable Diffusion’s computational cost, biases, and quality.

  • Problem

    Raster-trained diffusion models generate images without SVG primitives, while direct SVG modeling would require specialized architectures and substantial captioned-SVG data.

  • Method

    VectorFusion optimizes SVG shape parameters through a differentiable renderer guided by score distillation from a pretrained diffusion model, using raster initialization to improve efficiency and fidelity.

  • Results

    VectorFusion generates iconography, pixel art, and line drawings, achieves greater quality than CLIP-based approaches, and improves caption consistency over the best Stable Diffusion-plus-LIVE baseline by +3.9% R-Prec without initialization or rejection sampling.

  • Takeaways & Limitations

    Pixel-space generative priors can be distilled into interpretable, editable SVG outputs without captioned-SVG datasets, supporting existing design workflows.

  • Takeaways & Limitations

    VectorFusion requires more computationally expensive generative-model forward passes and remains limited by Stable Diffusion’s dataset biases and quality.

Abstract

from arXiv · show

Diffusion models have shown impressive results in text-to-image synthesis. Using massive datasets of captioned images, diffusion models learn to generate raster images of highly diverse objects and scenes. However, designers frequently use vector representations of images like Scalable Vector Graphics (SVGs) for digital icons or art. Vector graphics can be scaled to any size, and are compact. We show that a text-conditioned diffusion model trained on pixel representations of images can be used to generate SVG-exportable vector graphics. We do so without access to large datasets of captioned SVGs. By optimizing a differentiable vector graphics rasterizer, our method, VectorFusion, distills abstract semantic knowledge out of a pretrained diffusion model. Inspired by recent text-to-3D work, we learn an SVG consistent with a caption using Score Distillation Sampling. To accelerate generation and improve fidelity, VectorFusion also initializes from an image sample. Experiments show greater quality than prior work, and demonstrate a range of styles including pixel art and sketches. See our project webpage at https://ajayj.com/vectorfusion .

1. Introduction

VectorFusion addresses the difficulty of generating editable, scalable SVG graphics from diffusion models trained on raster images. It guides differentiable vector rendering with pretrained diffusion knowledge and improves caption coherence through optimization.

  • Motivation: SVGs represent visual concepts with editable primitives, scale to arbitrary resolutions, and remain compact, but designing them requires professional design-tool expertise.SVGs commonly use Bézier curves, polygons, circles, lines, and background colors.
  • Motivation: Web-scale captioned datasets are predominantly rasterized, so existing diffusion models generate finite-resolution images without primitive or layer decompositions.Direct SVG diffusion would require variable-length hierarchical architectures and substantial captioned-SVG data collection.
  • Problem: A raster-first pipeline often produces complex or caption-incoherent images that are difficult to approximate with simple vectors.LIVE vectorization minimizes image discrepancy without using the caption, which can discard information and reduce semantic coherence.
  • Approach: VectorFusion incorporates the pretrained text-to-image diffusion model into an optimization loop combining a differentiable vector renderer with score distillation sampling.The diffusion model supplies a signal for improving the rasterized output while SVG shape parameters are iteratively refined.
  • Contributions: The method extends score distillation to latent diffusion, initializes near raster samples, adds SVG-specific path reinitialization, and evaluates primitive choices for style control.Experiments target iconography, pixel art, and line drawings, with greater quality than CLIP-based approaches.

2. Related Work

Prior vector-generation methods use pretrained vision-language or image models with optimization, but raster diffusion models cannot directly produce vector graphics. VectorFusion instead uses a generative diffusion model as the guiding prior.

  • Vision-language guidance: VectorAscent and CLIPDraw optimize CLIP image-text similarity to generate vector graphics from text prompts.StyleCLIPDraw adds image conditioning through an auxiliary VGG16 style loss.
  • Vision-language guidance: Arnheim parameterizes SVG paths with a neural network, while CLIP-CLOP uses an evolutionary approach to create image collages.These approaches belong to a broader family of optimization-based vector-generation methods.
  • Diffusion models: Text-to-image diffusion systems such as Stable Diffusion produce high-fidelity raster images but cannot be directly transformed into vector graphics.Their training data and output representation remain pixel-based.
  • Image-to-vector methods: LIVE iteratively optimizes closed Bézier paths with a differentiable rasterizer to generate vectors from input images.VectorFusion extends this image-vectorization setting toward text-conditioned generation.
  • Diffusion priors: Diffusion priors have also guided zero-shot image-to-image tasks and text-to-3D synthesis, motivating their transfer to vector graphics.DreamFusion’s score distillation sampling provides the direct inspiration for using pixel-space diffusion as a transferable prior.

3. Background

The paper represents vector graphics as differentiable primitives, explains diffusion and latent diffusion as denoising models, and uses score distillation to optimize non-raster representations with a pretrained pixel-space prior.

  • Vector representations: Vector graphics are parameterized by paths whose control points, fill and stroke properties, and widths determine rendered shapes.Closed Bézier curves are used most often, while square shapes and unclosed Bézier curves support pixel art and line art.
  • Vector representations: DiffVG rasterizes SVG primitives while differentiating the rendered image with respect to path parameters.This makes image-space guidance usable for optimizing vector graphics, although raster-to-vector transformations can be lossy.
  • Diffusion models: Diffusion models learn data distributions by progressively denoising samples from a known prior, using weighted denoising objectives during training.A denoising autoencoder predicts noise content at sampled timesteps, and test-time sampling iteratively reduces the noise level.
  • Conditional diffusion: Text conditioning improves caption coherence through classifier-free guidance, which scales conditional outputs while requiring an additional unconditional forward pass.Conditional diffusion can otherwise produce caption-incoherent results because web datasets are weakly labeled.
  • Latent diffusion: Latent diffusion reduces computational cost by training in a compact latent space and decoding latent outputs into high-resolution raster images.Stable Diffusion is an open-source text-to-image model based on latent diffusion.
  • Transfer across representations: Before this approach, pretrained image diffusion models could generally generate only rasterized images, whereas encoder-based methods transferred more readily across modalities.CLIP and VGG16 could guide test-time optimization despite not being generative models.
  • Score distillation sampling: Score Distillation Sampling turns sampling into optimization by using gradients from a frozen pixel-space diffusion prior to update a differentiable image parameterization.It requires access to the pretrained pixel-space prior but not to a prior over the optimized parameter space.

4. Method: VectorFusion

VectorFusion converts text-conditioned raster diffusion knowledge into SVGs by combining image initialization, differentiable vector rendering, and latent score distillation. The method refines vector paths for caption coherence while supporting multiple abstraction styles through controllable primitives.

  • Pipeline: VectorFusion first samples a raster image, traces it into SVG paths, then refines the SVG with latent score distillation.The pipeline uses Stable Diffusion, staged path optimization through DiffVG, and a latent SDS refinement stage.
  • Baseline limitation: The baseline can lose caption information because LIVE optimizes an L2 image approximation without conditioning on the caption.This can produce SVGs that are visually close to the raster sample but no longer coherent with the text.
  • Optimization: Latent SDS rasterizes paths, applies augmentations, encodes the image, computes diffusion guidance in latent space, and backpropagates through the renderer to update paths.VectorFusion adapts SDS to latent diffusion models such as Stable Diffusion and treats the differentiable rendering pipeline as the optimized image generator.
  • Regularization: Path reinitialization removes low-opacity or tiny paths and recreates them to encourage more diverse and detailed vector images.The procedure periodically reinitializes paths whose fill-color opacity or area falls below a threshold.
  • Stylization: Constraining SVG primitives controls abstraction and style, enabling iconographic flat art, pixel art, and sketch-based line drawings.The paper studies how primitive choices and path counts affect the resulting visual style and detail.
  • Stylization: Using fewer paths produces simpler, flatter icons, whereas more paths create details and complex highlights.The comparison evaluates 16, 64, and 128 paths under Stable Diffusion initialization and SDS finetuning.

5. Experiments

VectorFusion is evaluated for caption consistency against raster, vectorization, and CLIP-based baselines, using automated CLIP metrics and qualitative comparisons. It performs competitively on quantitative consistency and produces coherent results across vector art, pixel art, and sketches.

  • 5.1. Experimental setup: The evaluation uses 128 captions and measures text-SVG coherence with CLIP Similarity and CLIP R-Precision because target SVG references are unavailable.The benchmark combines captions from prior text-to-SVG and text-to-image work.
  • 5.2. Evaluating caption consistency: LIVE conversion reduces caption consistency relative to Stable Diffusion raster samples, while VectorFusion improves over the strongest SD+LIVE baseline by 3.9% R-Prec.The comparison evaluates raster samples, automatic vectorization, and VectorFusion under different initialization and rejection-sampling settings.
  • 5.2. Evaluating caption consistency: With Stable Diffusion initialization and rejection sampling, VectorFusion matches or exceeds the best LIVE baseline by 15.7% R-Prec and improves over CLIPDraw by 4.0% with reranking.The method remains competitive with CLIPDraw even without using CLIP, showing a 0.8% OpenCLIP R-Prec advantage in that setting.
  • 5.3. Comparison with CLIP-based approaches: VectorFusion produces clearer and cleaner vector graphics than CLIP-based methods, while latent SDS further improves vibrancy and clarity over tracing Stable Diffusion samples.The comparison attributes the qualitative difference to incorporating a generative prior and optimizing paths with latent SDS.
  • 5.4. Pixel art generation: VectorFusion generates coherent pixel art that adds finer details and a characteristic pixel style beyond L1-pixelated Stable Diffusion samples.Figure 7 describes improvements despite the low-resolution 32x32 constraint.
  • 5.5. Sketches and line drawings: VectorFusion generates recognizable sketches from scratch, including complex multi-object scenes, while capturing caption semantics despite distractor style terms.The examples include ignoring terms such as “watercolor” and “Brightly colored” when they are irrelevant to sketches.

6. Discussion

VectorFusion transfers pretrained pixel-space diffusion knowledge to editable, scalable SVG generation without captioned SVG datasets. Its scope is constrained by computational cost and by Stable Diffusion’s dataset biases and image quality.

  • 6. Discussion: VectorFusion generates abstract SVG representations that can be used in existing design workflows without access to captioned SVG datasets.The discussion presents the method as a reference point for systems with interpretable and editable outputs.
  • 6. Discussion: VectorFusion requires more computationally expensive generative-model forward passes than contrastive approaches.The limitation is attributed to the generative model’s increased capacity.
  • 6. Discussion: VectorFusion is limited by Stable Diffusion’s dataset biases and quality, although improvements in text-to-image models are expected to improve it as well.The stated boundary follows the dependence of VectorFusion on Stable Diffusion.

B. Ablation: Reinitializing paths

VectorFusion periodically reinitializes paths that become too faint or small, encouraging broader path usage during optimization. This improves caption consistency for both random SDS initialization and LIVE-traced initialization.

  • Caption consistency: Reinitialization improves R-Precision by 3.0% when optimizing random paths with SDS.
  • Caption consistency: Reinitialization improves R-Precision by 12.5% when VectorFusion starts from a LIVE-traced sample.

C. Ablation: Number of paths

Increasing the number of SVG paths generally improves caption consistency and expressivity, but with diminishing returns. The study selects 64 paths as a practical balance between coherence, memory, and computation.

  • Caption consistency: More paths generally improve caption consistency, but the gains show diminishing returns.
  • Visual complexity: Fewer paths produce simpler, flatter icons, whereas more paths add details and complex highlights.
  • Caption consistency: 64 paths are sufficient to express and optimize coherent SVGs while balancing performance against time and memory constraints.The table describes greater expressivity and caption coherence with more paths, alongside increased memory and time complexity.

D. Ablation: Number of rejection samples

Rejection sampling substantially improves Stable Diffusion raster outputs, but vectorization reduces that coherence advantage. VectorFusion remains robust to initialization quality and needs only a few samples for high SVG-caption coherence.

  • Caption consistency: After LIVE vectorization, Stable Diffusion coherence falls by 10–15% in OpenCLIP H/14 R-Precision, despite more rejection samples improving the raster baseline.
  • Caption consistency: VectorFusion remains robust to rejection-sample count, with 1–4 Stable Diffusion samples sufficient for high SVG-caption coherence.
  • Baseline comparison: Increasing rejection samples improves Stable Diffusion and SD+LIVE caption consistency, while VectorFusion retains performance at K=20.
  • Baseline comparison: VectorFusion with no rejection sampling outperforms the SD+LIVE baseline even with 20 rejection samples.

E. Pixel Art Results

For pixel art, directly pixelating Stable Diffusion samples often produces blurry or unrealistic results. SDS finetuning with an L2 saturation penalty improves caption consistency and supports more characteristic pixel-art outputs.

  • Pixel art construction: Simply pixelating the best Stable Diffusion sample often produces unrealistic pixel art because the source lacks a regular pixel grid.The resulting images can appear blurry rather than characteristic of pixel art.
  • Pixel art evaluation: The ablation compares CLIP-based and SDS-based optimization and evaluates the saturation penalty as a factor in visual quality.
  • Pixel art construction: SDS finetuning with an additional L2 saturation penalty improves OpenCLIP R-Precision by 10.2%.
  • Style control: VectorFusion supports multiple styles through different primitive sets and optimization parameters, including iconographic art, pixel art, and sketches.
  • Pixel art representation: Pixel art is represented using a fixed 32×32 grid of square polygons whose RGB colors and opacities are initialized randomly.The square coordinates are not optimized.

F.2. Data Augmentation

VectorFusion uses distinct augmentation and optimization settings across training variants and visual styles. For pixel art, lower learning rates and an L2 saturation penalty reduce oversaturation artifacts.

  • Augmentation settings: SDS trained from scratch uses perspective and crop augmentation, while SD + LIVE + SDS does not use data augmentations.The rasterizer renders at 600x600, applies a perspective transform with probability 0.7, and then takes a random 512x512 crop.
  • Optimization settings: VectorFusion runs with batch size 1 on a single low-end GPU requiring at least 10 GB of memory.On an NVIDIA RTX 2080ti, SD + LIVE + SDS takes 10-20 minutes per SVG.
  • Style-specific optimization: For sketches and iconography, control-point learning rates warm from 0.02 to 0.2, then cosine-decay to 0.05.Fill colors use a 20× lower learning rate, while the solid background uses a 200× lower learning rate.
  • Style-specific optimization: For pixel art, the learning rate warms from 0.00001 to 0.0001 over 1000 iterations, alongside a weighted L2 saturation penalty with weight 0.05.Both the lower learning rate and the L2 penalty reduced oversaturation artifacts.
Loading 2211.11319v1…