PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

Sehwan Park, Taehoon Kim, Geonhee Han, Dohyun Kim, Seung Wook Kim, Paul Hongsuck Seo

arXiv:2609.14657v1cs.CVcs.AI

TL;DR

Existing SVG generation misses hierarchical semantic organization needed for intuitive manipulation and downstream reasoning; post-hoc recovery is bounded by paths that may entangle objects or leave occluded regions unrecoverable. A multi-role VLM recursively decomposes scenes, grounds semantic concepts in pixel space, recovers occluded geometry, and assembles components into grouped SVGs. Native predicted structures surpass optimal post-hoc groupings of existing baselines in grouping quality and functional editability while preserving state-of-the-art visual fidelity.

  • Problem

    Existing SVG generation misses hierarchical semantic organization needed for intuitive manipulation and downstream reasoning; post-hoc recovery is bounded by paths that may entangle objects or leave occluded regions unrecoverable.

    Source: §1 · p. 2

  • Method

    A multi-role VLM recursively decomposes scenes, grounds semantic concepts in pixel space, recovers occluded geometry, and assembles components into grouped SVGs.

    Source: §1 · p. 2

  • Results

    Native predicted structures surpass optimal post-hoc groupings of existing baselines in grouping quality and functional editability while preserving state-of-the-art visual fidelity.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The benchmark and sub-component metrics make semantic grouping and functional editability explicit evaluation targets beyond whole-image fidelity.

    Source: §1 · p. 2

    On the 203-image benchmark, processing averaged 255 seconds per image, with 94.9% spent on sequential inpainting inference.

    Source: §6 · p. 10

Abstract

from arXiv · show

While Vision-Language Models (VLMs) excel at visual reasoning, generating structured, editable Scalable Vector Graphics (SVG) remains a fundamental challenge. Existing pipelines predominantly yield flat, semantically agnostic collections of paths, where editing a single object requires manually identifying its constituent paths. To address this, we propose a VLM-driven agentic framework for semantic compositional SVG generation. Our pipeline recursively parses visual scenes into semantic and geometric hierarchies via top-down decomposition, visual grounding, and prompt-driven amodal occlusion recovery, ensuring each component is geometrically complete. Furthermore, we introduce the Semantic SVG Benchmark with human-annotated semantic groups and novel sub-component metrics (Semantic Recall/Precision, PERE) to explicitly evaluate structural compositionality and functional editability. Experiments show that our natively predicted structures surpass the upper bounds of existing flat-generation methods in both grouping quality and editability, while maintaining state-of-the-art visual fidelity.

1 Introduction

Existing SVG methods commonly produce flat paths without the semantic organization needed for intuitive editing. The paper proposes hierarchical scene parsing and a benchmark for evaluating semantic grouping and editability.

1 Introduction

  • Flat, semantically agnostic outputs leave users to identify constituent paths manually, while post-hoc grouping is bounded by entangled paths and unrecoverable occlusions.These constraints limit intuitive manipulation and downstream reasoning.Source: §1 · pp. 1–2
  • The proposed pipeline recursively decomposes scenes, grounds concepts in pixel space, recovers occluded geometry, and assembles complete, independently manipulable components.The framework uses a multi-role VLM to produce the semantic structure.Source: §1 · p. 2
  • The Semantic SVG Benchmark adds human-annotated semantic groups and metrics for semantic grouping and functional editability beyond whole-image fidelity.The metrics include Semantic Recall and Precision and Post-Edit Rendering Error.Source: §1 · p. 2
  • The pipeline outperforms existing flat-generation methods even when they receive optimal post-hoc grouping.This establishes a baseline for the proposed task.Source: §1 · p. 2

2 Related Work

Prior SVG-generation approaches produce flat representations, and existing evaluation focuses on whole-image similarity. The paper distinguishes its structural parsing approach from both these methods and general tool-augmented multimodal agents.

2 Related Work

  • Optimization-based and VLM-driven SVG methods produce flat outputs, while post-hoc grouping cannot subdivide entangled paths or recover occluded geometry.Existing benchmarks also rely on whole-image similarity rather than part-level metrics.Source: §2 · p. 2
  • The paper introduces human-annotated semantic groups, Semantic Recall/Precision, and PERE to assess semantic grouping and functional editability.These metrics address the absence of part-level evaluation in prior benchmarks.Source: §2 · p. 2
  • Unlike agents that use tools as auxiliary modules, this framework uses a VLM in a structural image-to-vector process with four cooperating roles and recursively invoked tools.Its tool outputs constitute the final SVG.Source: §2 · p. 2
  • The system overview shows recursive decomposition and grounding, followed by occlusion recovery for occluded leaves and back-to-front SVG assembly.The resulting nested group hierarchy mirrors the semantic tree.Source: Figure 2 · p. 3

3 Proposed Method

The method treats image-to-SVG generation as multimodal semantic parsing: a VLM builds a hierarchy of visual components, which is grounded, completed where occluded, and assembled as nested SVG groups.

3 Proposed Method

  • The framework formulates SVG generation as parsing a visual scene into a semantic and geometric hierarchy rather than generating a flat vector sequence.The hierarchy is represented as a tree whose nodes correspond to visual components and part-of relationships.Source: §3 · p. 3, §3.1 · p. 3

3.1 Problem Formulation: Semantic Compositional SVG Generation

  • The semantic tree represents visual components and their part-of relationships; intermediate nodes group children, while leaves represent atomic components rendered from vector primitives.The final SVG nests primitives according to background-to-foreground order.Source: §3.1 · p. 3

3.2 Top-Down VLM-Driven Decomposition

  • The VLM recursively decomposes non-atomic nodes into back-to-front child labels and boxes; SAM grounds them as pixel masks, and a residual judge adds missed concepts or discards noise.Expansion continues until every branch terminates in leaf nodes.Source: §3.2 · pp. 3–4

3.3 Prompt-Driven Semantic Completion

  • For each leaf, an occlusion assessor decides whether completion is needed; occluded leaves receive an estimated amodal polygon and inpainting to recover hidden geometry.The inpainting mask is the amodal polygon minus the visible SAM mask.Source: §3.3 · p. 4

3.4 Semantic Markup Assembly

  • An external vectorizer converts recovered leaf images into primitives, then a bottom-up pass nests labeled SVG groups to mirror the semantic hierarchy.The generated labels serve as group identifiers.Source: §3.4 · p. 4

3.5 Extension to Text-to-SVG

  • For text-to-SVG, a text-to-image model first synthesizes a raster image, which enters the same semantic decomposition and grouping pipeline as image-to-SVG.Source: §3.5 · p. 4

4 The Semantic SVG Benchmark

The Semantic SVG Benchmark supplies recursively annotated semantic hierarchies and evaluates grouping quality and post-edit geometric integrity, rather than whole-image fidelity alone. Its protocols compare predicted groups with annotated nodes and test whether deleting an object leaves the remaining scene intact.

4 The Semantic SVG Benchmark

  • The 203-sample benchmark combines prior-work and web SVGs, retains clearly decomposable scenes with substantial occlusion, excludes color-clustered outputs, and annotates labeled semantic groups recursively.The semantic trees have maximum depth 2, minimum depth 1, and mean depth 1.02.Source: §4 · pp. 4–5

4.1 Evaluating Semantic Grouping

  • Group agreement uses pixel-level MSE, where lower is better, or DINO similarity, where higher is better, computed over the union bounding box of the renderings.Source: §4.1 · p. 5

4 The Semantic SVG Benchmark

  • The benchmark covers 203 test samples, 535 semantic leaves, and 4,029 raw paths, with near-complete inter-annotator grouping agreement.Agreement was 0.0142 (MSE) and 0.9666 (DINO); disagreements mainly concerned background or shadow path membership.Source: §4.1 · p. 5, §4 · p. 5

4.1 Evaluating Semantic Grouping

  • For flat baselines, the protocol selects path subsets per ground-truth node; exact GT-conditioned search upper-bounds grouping quality, and a greedy algorithm approximates it in practice.Each path is an atomic grouping unit, and candidate groups jointly render subsets of paths.Source: §4.1 · p. 5
  • Semantic Recall matches each ground-truth node to its best-reconstructing candidate group, while Semantic Precision scores each predicted group against its best-matching ground-truth node.Precision is evaluated only for predicted groups, because the optimal candidate set is constructed per ground-truth node.Source: §4.1 · p. 5

4.2 Evaluating Editability and Geometric Integrity

  • Post-Edit Rendering Error (PERE) compares scenes after deleting matched occluding objects; high error indicates unrelated paths were grouped or occluded geometry was not recovered.The protocol targets front objects from intersecting depth-1 ground-truth groups and compares the resulting images over the full frame.Source: §4.2 · p. 6

5 Experiments

Experiments evaluate semantic grouping, editability, and visual fidelity for Image-to-SVG, alongside generation quality and text alignment for Text-to-SVG. Native predicted groups outperform baselines’ optimal post-hoc groupings on grouping quality and editability, with competitive whole-image fidelity.

5.1 Experimental Settings

  • Image-to-SVG evaluation uses the Semantic SVG Benchmark, while Text-to-SVG uses MMSVG-Bench metrics; experiments include SVG generators, VTracer, and prompted general-purpose VLMs.The pipeline uses Gemini-3-flash by default, with SAM 3 for grounding, Flux-Fill for occlusion recovery, and VTracer for vectorization.Source: §5.1 · p. 6

5.2 Results

  • Native predicted groups surpass every baseline’s optimal configurations across grouping metrics, while whole-image fidelity remains comparable to VTracer and ahead of generative baselines.Ours (optimal) achieves Recall of 0.0091 in MSE and 0.9571 in DINO; predicted groups also exceed VTracer’s Recall values of 0.0227 vs. 0.0174 in MSE and 0.9392 vs. 0.9487 in DINO.Source: §5.2 · p. 6, Table 1 · p. 7
  • Figure 3 shows Ours (pred) closely matching ground truth in object isolation and background completeness, unlike baselines that leave holes or merge similarly colored regions.The figure compares whole images, isolated objects, and residual backgrounds; baseline objects use optimal post-hoc path selection.Source: §5.2 · p. 7, Figure 3 · p. 7
  • All edits on the method’s SVGs were completed roughly 1.9× faster than on VTracer outputs; removal yielded MSE of 0.006 vs. 0.023.The study tested moving and removing objects, with group selection manipulating all paths in a group at once.Source: §5.2 · p. 8, Table 2 · p. 8
  • Gemini-3-flash performs best across the tested backbones, while enabling occlusion recovery improves every metric, reducing Grouping F1 in MSE by 41.1% and PERE by 31.6%.GPT-5 often under-segments; disabling inpainting leaves a mask-and-crop system that extracts only visible pixels.Source: §5.2 · p. 8, Table 4 · p. 9
  • Both Text-to-SVG variants substantially outperform baselines; FLUX achieves the best CLIP, HPSv2, and Aesthetic scores, while OmniSVG’s lowest FID is attributed to training-data overlap.Visual comparisons show faithful scene synthesis and structurally complete occluded geometry behind foreground objects.Source: §5.2 · p. 9, Table 5 · p. 9

6 Conclusion

The work recasts SVG generation as semantic composition, using a multi-role VLM to create geometrically complete, language-aligned components and a benchmark for grouping and editability. Native structures outperform optimal post-hoc groupings in grouping quality and editability while preserving state-of-the-art visual fidelity; computational cost is the main limitation.

6 Conclusion

  • Native structures surpass optimal post-hoc groupings in grouping quality and functional editability while preserving state-of-the-art visual fidelity.The paper introduces the human-annotated Semantic SVG Benchmark and Semantic Recall/Precision and Post-Edit Rendering Error to evaluate these properties.Source: §6 · p. 9
  • A multi-role VLM recursively decomposes scenes into geometrically complete, language-aligned components for semantic compositional SVG generation.The work also introduces the human-annotated Semantic SVG Benchmark and sub-component metrics for grouping and editability.Source: §6 · p. 9
  • Computational cost is the framework’s main limitation, with 255 s average end-to-end latency per image.The authors report that sequential FLUX.1-Fill-dev inference accounts for 94.9% of latency and suggest asynchronous processing across stages as a mitigation.Source: §6 · p. 10

A.1 Comparison with Existing SVG Datasets

Existing SVG benchmarks cover diverse domains but lack semantic group annotations, while this benchmark adds path-level groups and component-level evaluation.

A.1 Comparison with Existing SVG Datasets

  • Existing SVG benchmarks cover diverse domains but lack semantic group annotations; this benchmark adds path-level groups, component-level grouping metrics, and PERE alongside whole-image MSE and DINO.The comparison includes SVG-Bench, MMSVG-Bench, SArena, and VectorGym.Source: §A.1 · p. 12

A.2 Source Data and Licenses

The benchmark combines filtered, reannotated samples from prior datasets with samples collected directly from public emoji and clipart projects, and adds its own semantic and occlusion annotations.

A.2 Source Data and Licenses

  • After cross-set deduplication and filtering, retained prior-benchmark samples came from SArena icon and illustration subsets and SVG-Bench’s SVG-Emoji test subset.These samples form the reannotated portion of the benchmark.Source: §A.2 · p. 12
  • Other samples were collected from public emoji and clipart repositories, and none were byte-identical to SVGs in the surveyed benchmark corpora.Sources include Openclipart, Twemoji, Noto Emoji, EmojiTwo, Fluent Emoji, and OpenMoji.Source: §A.2 · p. 12
  • All sources permit research use and were individually license-verified; OpenMoji requires attribution upon redistribution, and semantic and occlusion annotations were created by the authors.The source data and licenses are summarized in Table 7.Source: §A.2 · p. 12

A.3 Benchmark Construction Process

The benchmark construction process filters for object-level partitionability, builds hierarchical semantic annotations, and manually validates primitive ownership. The Text-to-SVG extension adapts two text-to-image models and feeds their generated rasters into the pipeline.

A.3 Benchmark Construction Process

  • Filtering excludes samples with paths spanning multiple semantic objects, indivisible compositions, no inter-object occlusion, or color-region rather than object-boundary vectorization.Object-level partitionability requires visual inspection because SVG markup alone cannot establish it.Source: §A.3 · p. 12
  • Annotators use a path-highlighting interface to assign paths to semantic groups and recursively decompose each scene until every branch reaches a leaf.Each leaf records a free-text label and its constituent paths; occluded paths are grouped by the object they define.Source: §A.3 · p. 12, Figure 5 · p. 12
  • A node is split only when each part has an independent semantic identity; dependent parts stay grouped, while semantically empty elements are absorbed or excluded from evaluation.Labels are free text rather than drawn from a fixed vocabulary.Source: §A.3 · pp. 12–13
  • Manual validation requires every drawable primitive to belong to exactly one leaf, though rendered leaves may overlap through occlusion; 203 SVGs passed filtering, annotation, and validation.Released samples include the raw SVG, decomposition tree, provenance, and license.Source: §A.3 · p. 13

B.1 Details on the Text-to-SVG Extension

  • FLUX.1-dev and SD3.5-medium were adapted for icon and illustration raster generation using rank-128 LoRA adapters trained on 32,000 rasterized MMSVG images.Text encoders, VAE, and remaining model layers were frozen.Source: §B.1 · p. 13
  • Inference uses the FLUX.1-dev checkpoint after one epoch and SD3.5-medium after two epochs, with 28 and 40 sampling steps, respectively.The corresponding guidance settings are 3.5 for FLUX.1-dev and CFG scale 4.5 for SD3.5-medium.Source: §B.1 · pp. 13–14

B.2 Evaluation Details

The evaluation compares predicted semantic groups with ground-truth components using independent nearest-neighbor matches, and tests editability by deleting occluding objects and measuring rendering error. Flat baselines’ optimal grouping is a greedy approximation to a per-component path-subset target, not a global partition or certified optimum.

Greedy Approximation of Optimal Grouping

  • Flat-baseline optimal grouping is a greedy approximation to a ground-truth-conditioned path-subset target, not a certified optimum or global partition.Subsets are optimized independently per ground-truth node and may overlap.Source: §B.2 · p. 14

Predicted Grouping and Bidirectional Matching

  • Semantic Recall and Precision use independent nearest-neighbor matches, and dataset-level scores macro-average per-image results rather than matching groups one-to-one.Recall matches each ground-truth node to its best predicted group; Precision matches each predicted group to its best ground-truth node.Source: §B.2 · p. 14

Post-Edit Rendering Error

  • PERE tests occlusion edits by deleting the foreground group and comparing edited SVGs with full-frame MSE, averaged across occlusion pairs and images.The benchmark includes 337 valid pairs across 203 images; each deletion starts from the original SVG.Source: §B.2 · pp. 14–15
  • Predicted-setting PERE matches each ground-truth occluder to the outermost labeled generated group with the lowest full-frame MSE.For optimal-setting PERE, the corresponding group is instead the MSE-selected subset of paths.Source: §B.2 · pp. 14–15

Relation to Whole-Image Fidelity

  • PERE is an end-to-end editability measure, not grouping quality alone: it is unnormalized by whole-image error, while optimal grouping remains sensitive to missing or incorrect primitives.For methods evaluated in both settings, the SVG is identical, so Semantic Recall and PERE differences reflect predicted groups versus GT-conditioned path optimization.Source: §B.2 · p. 15

C Evaluation on Image-to-SVG with MMSVG-Bench

On MMSVG-Bench, the method achieves the best scores across all four reported whole-image fidelity metrics. Because the benchmark lacks semantic-group annotations, it cannot assess grouping quality or editability.

C Evaluation on Image-to-SVG with MMSVG-Bench

  • The method achieves the best scores across all four whole-image fidelity metrics on 300 MMSVG-Bench samples, but grouping quality and editability cannot be measured without semantic-group annotations.The evaluation is limited to whole-image fidelity; despite decomposing scenes and reassembling per-component vectorizations, the method attains the highest fidelity scores.Source: C Evaluation on Image-to-SVG with… · p. 15

D More Qualitative Results

Additional examples show semantic-object isolation and recovery of occluded backgrounds across image-to-SVG and text-to-SVG scenes. The accompanying prompts specify decomposition criteria and conventions for component labels and ordering.

D More Qualitative Results

  • Additional image-to-SVG and text-to-SVG examples show isolated semantic objects and residual backgrounds, with the authors describing robust entity isolation and amodal recovery across diverse scenes.The image-to-SVG figures show whole images, isolated objects, and residual backgrounds; the text-to-SVG examples include four prompts.Source: D More Qualitative Results · p. 15, Figure 9 · p. 20

E.1–E.10 Prompt Examples

  • Prompt examples cover region-decomposition decisions, residual and inpainting judges, polygon prediction variants, inpainting selection, and image- and text-to-SVG baseline prompts.The polygon prompts include category-aware, structured-reasoning, and geometric-strict variants.Source: §E.1 · p. 16, §E.2 · p. 16, §E.3 · p. 16

E.10 Text-to-SVG Prompt for the VLM Baselines

  • The decomposition prompt defaults to atomic regions and splits only when at least two named sub-parts each occupy 25% or more of the parent box and visibly occlude one another.At depth two, non-atomic splits are rare and require all checks plus an obvious split; depth three or greater is blocked by code.Source: §E.10 · p. 21
  • The prompt treats tools and characters as atomic bodies, groups repeated small shapes rather than enumerating instances, and excludes decorative details and shading as separate parts.It also forbids placeholder labels such as “background” and “residual”; unnamed pixels are absorbed by the parent mask.Source: §E.10 · p. 21
  • Components are listed back-to-front, with backgrounds and containers before foregrounds and contents, so occlusion order informs layer reconstruction.The prompt also specifies labels as plain noun phrases and tight bounding boxes in normalized coordinates of the original image.Source: §E.10 · p. 21
Loading 2609.14657v1…