PaperlayerView exact revision on arXiv

Source-linked AI summary

DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance

Peiying Zhang, Nanxuan Zhao, Matthew Fisher, Yiran Xu, Jing Liao, Difan Liu

arXiv:2512.10894v1cs.CV

TL;DR

SVG generation methods based on text tokens lack visual guidance and are limited in their use of image data. DuetSVG jointly generates image and SVG tokens, then uses selected image predictions to guide SVG decoding. The paper reports stronger performance than existing methods across benchmarks and notes that fine image details and color variation remain challenging.

  • Problem

    Text-only SVG generation lacks visual guidance during decoding and is restricted to relatively small SVG datasets, limiting generalization.

    Source: §1 · p. 2

  • Method

    DuetSVG jointly generates image and SVG tokens, trains on image and SVG tasks, and uses internal image predictions to guide SVG decoding.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    DuetSVG consistently outperforms existing text-to-SVG baselines across metrics and achieves much higher quality than state-of-the-art methods on multiple benchmarks.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The unified framework supports SVG generation and editing tasks while leveraging image datasets and native visual guidance.

    Source: §1 · p. 2, §6 · p. 9

    For input images with very fine details and rich color variation, generated SVGs may miss small structures and show mild color shifts.

    Source: C. Limitation · p. 13

Abstract

from arXiv · show

Recent vision-language model (VLM)-based approaches have achieved impressive results on SVG generation. However, because they generate only text and lack visual signals during decoding, they often struggle with complex semantics and fail to produce visually appealing or geometrically coherent SVGs. We introduce DuetSVG, a unified multimodal model that jointly generates image tokens and corresponding SVG tokens in an end-to-end manner. DuetSVG is trained on both image and SVG datasets. At inference, we apply a novel test-time scaling strategy that leverages the model's native visual predictions as guidance to improve SVG decoding quality. Extensive experiments show that our method outperforms existing methods, producing visually faithful, semantically aligned, and syntactically clean SVGs across a wide range of applications.

1. Introduction

SVGs offer editable, resolution-independent graphics, but producing them is complex, and text-only generation lacks visual guidance for coherent rendering and broader training data. DuetSVG jointly generates image and SVG tokens, uses visual guidance during decoding, and reports stronger quality than state-of-the-art methods across multiple benchmarks.

1. Introduction

  • Text-only SVG models lack visual guidance, so small coordinate errors can cause catastrophic rendering failures; their limited SVG training data also constrains generalization.They cannot leverage the larger supply of raster-image datasets described in the passage.Source: §1 · p. 2
  • DuetSVG jointly generates image and SVG tokens, using its native image predictions as decoding guidance and enabling training on both image and SVG datasets.The paper also describes the framework as supporting text-to-SVG, image-to-SVG, SVG completion, and editing.Source: §1 · p. 2
  • DuetSVG achieves much higher quality than state-of-the-art methods on multiple benchmarks.The introduction presents native visual guidance, image-dataset training, and efficient test-time scaling as contributions.Source: §1 · p. 2

2. Related Work

Prior SVG-generation work spans optimization-based reconstruction, learned vector generation, and unified multimodal models. The section describes limitations in efficiency, structure, generalization, and SVG-specific training that motivate DuetSVG’s unified approach.

2.1. Optimization-based SVG Generation

  • Optimization-based vectorization can reproduce overall appearance but may yield redundant paths and imprecise control points; VLM- or diffusion-guided optimization can take tens of minutes per SVG and still produce hard-to-edit paths.The latter methods are described as producing fragmented paths, inconsistent topology, and redundant control points.Source: §2.1 · p. 2

2.2. Learning-based SVG Generation

  • Learning-based methods model SVGs with geometric primitives or tokenized scripts, but raster-to-vector pipelines can generalize poorly, while text-token models lack visual guidance and are limited by small SVG datasets.The section contrasts DuetSVG’s end-to-end co-generation of image and SVG tokens, which enables text-image data use and visual grounding during decoding.Source: §2.2 · p. 2

2.3. Unified Multimodal Generation

  • Unified multimodal models generate or understand multiple modalities, yet state-of-the-art VLMs still fail to produce high-quality vector graphics without SVG-specialized training.The section notes both fully autoregressive and autoregressive-diffusion fused paradigms.Source: §2.3 · pp. 2–3

3. Method

DuetSVG defines SVG generation as conditional mixed-modality token prediction, using one model to generate image and SVG tokens. Its training combines text-to-image pretraining with multi-task fine-tuning, while inference uses image-guided SVG resampling; SVG-Hub-1M supplies cleaned, standardized SVGs with rich captions.

3.1. Task Definition

  • Given multimodal input, DuetSVG predicts a mixed target sequence of image tokens followed by SVG tokens under a unified next-token objective.The input may include text prompts, images, and SVG code; the image modality captures appearance while SVG tokens represent geometry and layer structure.Source: §3.1 · p. 3

3.2. Unified Multimodal SVG Generation Model

  • The architecture aligns image features with text embeddings and uses a unified autoregressive transformer with separate heads for image-token and SVG-token prediction.It accepts text, SVG, and image inputs, using an understanding encoder for image semantics and a generation encoder for discrete visual tokens.Source: §3.2 · p. 3, Figure 2 · p. 3
  • Large-scale text-to-image pretraining on real and synthetic data initializes DuetSVG with visual and semantic priors for SVG generation.The stage targets clean, visually appealing images with clear geometric primitives and flat colors, and is reported to improve generalization to complex and out-of-distribution prompts.Source: §3.2 · pp. 3–4

3.3. Test-Time Scaling with SVG Resampling

  • Test-time scaling selects a CLIP-scored image candidate, then generates SVG chunks and accepts each update only when its rendered image’s LPIPS distance to that candidate does not increase.Because image-token sequences are shorter than SVG-token sequences, candidate selection is relatively efficient; rejected chunks are resampled, with a limit on rejections.Source: Figure 3 · p. 4, §3.3 · p. 4

3.2. Unified Multimodal SVG Generation Model

  • Multi-task fine-tuning trains on text-to-image, text-to-SVG, and image-to-SVG, arranging SVG-task targets as image tokens followed by SVG tokens so visual tokens guide decoding.Augmentations modify SVG geometry, color, or paths; 10% random text and image dropout enables classifier-free guidance, and optional downstream fine-tuning supports tasks such as completion.Source: §3.2 · p. 4

3.4. SVG-Hub Dataset

  • SVG-Hub-1M is curated from public SVG sources by removing duplicates, auto-vectorized files, and blank renderings, then standardizing scripts and pairing each SVG with captions at three levels of detail.The tokenization normalizes the canvas to an 800×800 viewBox, restricts the command vocabulary, and quantizes coordinates while retaining gradients and group transformations.Source: §3.4 · pp. 4–5

4. Experiments

DuetSVG is evaluated on two benchmarks across text-to-SVG and image-to-SVG tasks, with comparisons against optimization-based and learning-based baselines. The experiments report stronger text-to-SVG results and examine the contributions of visual output, T2I pretraining, and test-time scaling.

4.1. Implementation Details

  • Training includes T2I pretraining followed by multi-task SFT, with the full process taking about 14 days on 64 NVIDIA-A100 GPUs.The SFT stage samples T2I, T2SVG, and I2SVG data at a 1:5:4 ratio; test-time scaling uses N = 3 and M = 3.Source: §4.1 · p. 5

4.2. Experiment Setup

  • The evaluation uses 9,000 SVG-Hub-5M test samples and 6,000 SArena-Icon samples, with vector- and image-level metrics for the two tasks.The baselines include optimization methods and proprietary and open-source VLMs; open-source VLMs are fine-tuned on SVG-Hub-5M for comparison.Source: §4.2 · p. 5

4.3. Comparisons

  • DuetSVG consistently outperforms all text-to-SVG baselines across metrics on both benchmarks.The comparisons cover the SVG-Hub-5M test set and SArena-Icon benchmark.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 6
  • Optimization-based methods can produce redundant paths and artifacts, while proprietary VLMs rely on simple primitives and fine-tuned open-source VLMs still miss complex semantics and geometry.DuetSVG instead produces visually appealing SVGs with clear structure and semantically rich content.Source: §4.3 · p. 6

4.4. Ablation Study

  • Removing visual output degrades metrics and semantic and structural coherence; removing T2I pretraining weakens visual appeal and SVG complexity.The ablation results attribute stronger visual and semantic priors to T2I pretraining.Source: §4.4 · p. 7, Table 3 · p. 8, Figure 6 · p. 8
  • The two-stage test-time scaling strategy with image-guided SVG resampling is more efficient than best-of-N and improves quality in both T2SVG and I2SVG.Without TTS, autoregressive decoding may introduce local geometric distortions on complex inputs.Source: §4.4 · p. 8, Table 3 · p. 8, Figure 6 · p. 8

5. Applications

After fine-tuning, DuetSVG supports SVG completion and instruction-based editing. The paper also presents its test-time scaling strategy as a more efficient alternative to best-of-N sampling.

SVG completion

  • For SVG completion, the model receives a partial SVG and its rendered image, then infers the complete image and SVG script.Training masks a random subset of paths; the resulting completions are described as coherent and visually appealing.Source: §5 · p. 8, Figure 8 · p. 8

Test-time scaling

  • Image-level candidate selection and SVG resampling achieve substantially lower sampling cost than best-of-N test-time scaling.The comparison is presented as a sampling-efficiency evaluation.Source: Figure 7 · p. 8

SVG editing

  • For semantic SVG editing, synthetic training pairs use an edited image and inverse instruction as inputs to reconstruct the original image and SVG script.The paper reports that the fine-tuned model can modify SVG content according to text instructions.Source: §5 · p. 8, Figure 8 · p. 8

6. Conclusion

DuetSVG jointly generates image and SVG tokens, using visual predictions to guide SVG decoding. The paper reports stronger SVG quality and reliability, and identifies larger unified multimodal models as future work.

6. Conclusion

  • DuetSVG generates image tokens and SVG tokens rather than treating SVGs as pure text.The framework combines large-scale T2I pretraining with multi-task SFT on T2I, T2SVG, and I2SVG.Source: §6 · p. 9
  • Its vision-aware test-time scaling uses internal visual predictions to guide SVG decoding and improve robustness and reliability.Source: §6 · p. 9
  • The authors identify larger unified multimodal models, such as Emu3.5, as a direction for future work.Source: §6 · p. 9

Supplementary Material

The supplementary material lists additional evaluations and implementation-related details.

A. Overview

  • Supplementary material covers generalization evaluation, additional comparisons and results, limitations, and the captioning pipeline.Source: A. Overview · p. 12

B.1. Generalization Evaluation

DuetSVG is evaluated for novelty and uniqueness on outputs sampled from text prompts, with nearest-neighbor examples used to illustrate the results.

  • Novelty means no SVG-Hub-5M match has CLIP cosine similarity of 0.98 or higher; Uniqueness counts samples appearing exactly once in the generated set.The paper renders each generated SVG and retrieves its nearest SVG-Hub-5M neighbor to assess Novelty.Source: §B.1 · p. 12
  • 99.5% Novelty and 99.8% Uniqueness are reported for 1,500 generated SVGs.The samples come from 500 prompts, with three SVGs generated per prompt using different seeds.Source: §B.1 · p. 12
  • The evaluation uses 500 randomly selected prompts and generates three SVGs per prompt with different seeds.Figure 9 shows generated SVGs alongside their nearest neighbors from SVG-Hub-5M.Source: §B.1 · p. 12

B.2. Additional Comparisons and Results

On SVG-Hub-1M, DuetSVG consistently outperforms prior approaches across metrics, while a two-stage image-to-SVG baseline faces a train-test mismatch. The section also includes qualitative comparisons and failure-case figures.

  • On SVG-Hub-1M, DuetSVG consistently outperforms prior approaches across metrics, and its performance improves substantially as dataset size increases.Other methods show only limited improvement on the text-to-SVG task as dataset size increases.Source: §B.2 · p. 12
  • Figure 9 presents generated SVGs alongside their nearest-neighbor samples.Source: Figure 9 · p. 12
  • The FLUX.1-to-Qwen3-VL-8B baseline faces a train-test mismatch because its synthesized intermediate images differ from the real SVG images used during I2SVG training.The resulting SVGs often show reduced alignment with the intermediate images; DuetSVG instead co-generates image and SVG tokens end to end.Source: §B.2 · p. 12, Figure 10 · p. 13
  • Figure 11 presents failure cases.Source: Figure 11 · p. 13

C. Limitation

The limitation section identifies missed small structures and mild color shifts for inputs with very fine detail and rich color variation. The captioning pipeline uses cross-model caption verification and provides templates at three levels of detail.

C. Limitation

  • Inputs with very fine details and rich color variation may yield SVGs that miss small structures or show mild color shifts.The authors propose investigating dynamic high-resolution processing, which adaptively increases vision-encoder patches, as a potential mitigation.Source: C. Limitation · p. 13

D. Details of the captioning pipeline

Loading 2512.10894v1…