PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Design

Shibo Yang, Yuqing Gao, Zipeng Liu

arXiv:2609.06116v1cs.HCcs.CV

TL;DR

Traditional SVG editing requires expertise and manual iteration, while LLM-based methods face targeting and controllability challenges. MM-SVGEdit combines direct manipulation with LLM editing through visual grounding followed by modification. The authors report stronger editing accuracy and controllability than direct generation, alongside reduced token use and response time.

  • Problem

    Traditional SVG tools impose expertise and manual-work burdens, while LLM-based editing can suffer from ambiguous target specifications and limited fine-grained control.

    Source: §1 · pp. 1–2

  • Method

    MM-SVGEdit uses a two-stage workflow—visual grounding followed by modification—with both stages supporting natural-language instructions and direct manipulation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across evaluations, the framework outperforms native direct generation in editing accuracy and controllability; Qwen3-32B accuracy improves from 76.1% to 87.3% (improvement of +11.2%), with token use and response time also reduced.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The system offers a structured UI-SVG editing workflow in which users can confirm target elements before localized modifications are applied.

    Source: §7 · p. 10

    The system is designed and trained primarily for SVG-based UIs, and performance on other domains or more complex SVG graphics remains unvalidated.

    Source: §6 · p. 10

Abstract

from arXiv · show

In the field of UI design, Scalable Vector Graphics (SVG) is widely used as a design medium. However, traditional SVG editing techniques have high entry barriers and require cumbersome manual iteration, while LLM-based editing solutions suffer from low accuracy and poor user controllability. To address these issues, we propose MM-SVGEdit, a multimodal-driven SVG editing approach that integrates traditional SVG editing and LLM-based methods. We introduce a two-stage strategy in which visual grounding is followed by modification. Both stages support two interaction modalities: natural language instructions and direct manipulation (mouse and keyboard). We trained and evaluated MM-SVGEdit on a self-constructed dataset of 14,476 question-answer pairs generated from UIs, covering 11 types of editing operations on both single and multiple UI targets. The results show that MM-SVGEdit improves SVG editing accuracy, efficiency, and user-perceived control while reducing token consumption and response time.

1 Introduction

Traditional SVG tools demand design or programming expertise and manual iteration, while LLM-based editing can struggle with precise targeting and user control. MM-SVGEdit combines direct manipulation and LLM editing in a two-stage, multimodal workflow.

1 Introduction

  • Manual SVG tools require substantial expertise and tedious operations, while text-only LLM editing can leave target selection ambiguous and outputs difficult to control.The paper focuses on structured, browser-renderable UI SVGs for localized, code-backed editing rather than replacing full-featured design environments.Source: §1 · pp. 1–2
  • MM-SVGEdit integrates mouse-and-keyboard editing with LLM-based editing so users can choose a modality for each task.Its two stages are visual grounding followed by modification, and both support natural-language instructions and direct manipulation.Source: §1 · p. 2
  • The approach uses Agent Locator for visual grounding and Agent Modifier for modification, trained or configured using a self-constructed UI-SVG dataset.The dataset and representative editing operations support systematic training and evaluation of both agents.Source: §1 · p. 2
  • The authors report improved editing accuracy, efficiency, and user-perceived control while keeping token consumption and response time within acceptable bounds.They characterize the added computational overhead as minimal.Source: §1 · p. 2

2 Related Work

Prior work spans manual SVG editing, AI-driven visual editing, and datasets for vector editing. The paper positions MM-SVGEdit against barriers in manual tools, limits in existing AI workflows, and the lack of UI-specific SVG editing data.

2.1 Traditional SVG Editing Technologies

  • Traditional desktop tools and development libraries depend on manual design or coding work, creating high entry barriers and cumbersome editing even for professionals.The paper describes tedious path editing, node adjustment, and code debugging as constraints on flexibility and efficient creation.Source: §2.1 · p. 2

2.2 AI-Driven SVG Editing Technologies

  • AI-driven work includes raster-image editing and SVG-specific methods, but much of it targets image manipulation, generation, or analysis rather than iterative UI-SVG editing.Chat2SVG combines structured SVG templates, raster targets, and SVG optimization [28]; Design2Code studies screenshot-to-code generation [22], not iterative vector editing.Source: §2.2 · pp. 2–3

2.3 Datasets for SVG Editing

  • Existing SVG datasets support general vector editing, but are not tailored to UI-specific tasks; the paper constructs a UI-SVG dataset from bitmap UI datasets with fine-grained annotations and instruction-action pairs.The related datasets include SAgoge [25], VectorEdits [13], SVGEditBench [16], and SVGenius [6]; bitmap UI resources such as Rico also lack direct support for precise SVG manipulation.Source: §2.3 · p. 3

3 Multimodal Editing System

MM-SVGEdit combines direct manipulation and LLM editing in a workflow that grounds target elements before applying structured modifications. Its agents and layered interface support human confirmation and multimodal interaction throughout editing.

3.1 Multimodal Scheme

  • The multimodal scheme combines manual and natural-language editing, with users able to switch or combine modalities and correct candidate selections before localized modifications.Natural language suits semantic or multi-element changes, while direct manipulation supports precise local refinements.Source: §3.1 · pp. 3–4

3.2 Agent Locator

  • Agent Locator maps an SVG and natural-language instruction to target element IDs, using a dedicated instruction-tuning dataset and a fine-tuned Qwen2.5-7B-Instruct model.The reported fine-tuned model outperforms its pretrained version and performs comparably to commercial models, especially on single-target grounding.Source: §3.2 · pp. 4–5

3.3 Agent Modifier

  • Agent Modifier separates intent parsing from execution through standardized function specifications and localized operations on the grounded target elements.The workflow generates calls, presents affected elements for user confirmation, then executes the confirmed operations; the specification constrains permitted parameters and operations.Source: §3.3 · p. 5

3.4 System Architecture and Interface

  • The four-layer architecture connects user interaction, intelligent reasoning, the editing engine, and data persistence in one system pipeline.It supports natural-language input and mouse operations, as well as independent or collaborative use of the two modalities.Source: §3.4 · p. 6, §3.4.1 · p. 6

3.4.1 Layers Design.

  • The layer design provides multimodal input and real-time feedback, with reasoning that maps instructions to SVG attributes and editing functions and an engine that supports LLM and direct-manipulation pathways.The persistence layer maintains SVG templates and component libraries and records operation logs and interaction history.Source: §3.4.1 · p. 6

3.4.2 System Interface.

  • The interface places multimodal interaction panels beside a central SVG editing and preview area, with preview and results controls above.The side panels expose natural-language and mouse/keyboard controls and element attributes; selected elements are highlighted in purple.Source: §3.4.2 · pp. 6–7, Figure 3 · p. 6

4 Dataset Construction

Dataset construction converts UI screenshots into editable SVGs, defines 11 editing-operation categories, and generates 14,476 instruction–UI pairs with target annotations. The resulting resources support model training and evaluation.

4.1 Convert to SVGs

  • Bitmap screenshots are converted into structured SVGs with extracted element attributes, reconstructed hierarchies, unique IDs, specification checks, and mobile-adaptive filtering.This pipeline addresses the lack of element-level editability in bitmap-based UI datasets.Source: §4.1 · p. 7

SVGs

  • SVG reconstruction accuracy exceeds 85% for medium-complexity UI scenarios.The constructed SVG dataset preserves original annotations, interface hierarchies, and unique element IDs.Source: SVGs · p. 7, §4.1 · p. 7

4.2 Define Editing Operations

  • The operation taxonomy covers 11 core UI editing categories, with detailed subcategories, examples, and scopes for single- and multi-target instructions.A UI graphic can accommodate as many as 38 distinct editing instructions.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Generate Ground Truths

  • GPT-4o generates instructions using element IDs as ground truth, followed by automated filtering and manual review, yielding 14,476 QA pairs across all 11 categories.Each sample links an SVG UI, an instruction targeting one or more elements, and the corresponding target element ID(s).Source: §4.3 · p. 8
  • A separate edited-SVG dataset contains 213 instructions with LLM generation and manual verification, supporting instruction-alignment, element-editing, and incremental-edit evaluation.Source: §4.3 · p. 8

5 Evaluation

The evaluation tests visual grounding and structured function calling for SVG editing. Results report stronger grounding and editing accuracy, lower token use and response time, and limits around element IDs and structured layouts.

5.1 Evaluation for Agent Locator

  • The grounding test uses 500 samples spanning diverse UI styles and all 11 editing categories; models receive an SVG and instruction and return the target element ID.Accuracy compares the predicted ID with the annotated ground-truth ID.Source: §5.1 · p. 8
  • 92% overall accuracy on mixed single- and multi-target instructions for fine-tuned Qwen2.5-7B; function calling beats direct editing across models (DeepSeek-V3.2: 97.2% vs 89.7%).Source: Table 2 · p. 9, §5.1 · p. 9, Table 3 · p. 9

5.2 Evaluation for Agent Modifier

  • Function calling makes editing less opaque and reduces incorrect element selection and unintended modifications, improving reliability and practical usability.Source: §5.2 · p. 9
  • Claude-3-Haiku function calling reduces token consumption from 2010 to 178 (91.1%) and response time from 18.8 to 2.6 seconds (85.9%).For Qwen3-32B, token consumption falls from 2517 to 706 (72.0%), and response time from 71.3 to 21.3 seconds (70.1%).Source: Table 4 · p. 9, §5.2 · p. 9
  • The pipeline relies on element IDs embedded in SVGs, and performance on irregularly arranged elements remains unverified.Without embedded IDs, additional preprocessing increases system complexity.Source: §5.2 · p. 9

6 Discussion

MM-SVGEdit combines visual grounding with structured function calling to improve editing accuracy, controllability, and efficiency. Its evaluation reports gains in grounding and editing performance, while the authors note that broader SVG domains remain unvalidated.

6 Discussion

  • The work addresses modal fusion, accuracy, inference efficiency, and controllability through a two-stage pipeline integrating visual-grounding fine-tuning and structured function calling.The authors report validation through quantitative and qualitative experiments.Source: §6 · p. 9
  • Grounding accuracy on mixed single- and multi-target instructions rises from 62% to 92%, surpassing all tested commercial LLMs.The model is Qwen2.5-7B, fine-tuned on the custom UI-oriented dataset.Source: §6 · pp. 9–10
  • The structured function-calling framework outperforms native direct generation across open-source and commercial LLMs, improving editing accuracy and controllability.Qwen3-32B rises from 76.1% to 87.3% (+11.2%).Source: §6 · p. 10
  • On lightweight models, response time falls by 85.9% and token consumption by 91.1%; large open-source models show 70% lower latency.The passage also reports corresponding token-consumption reductions for large open-source models without specifying their values.Source: §6 · p. 10
  • The system is designed and trained primarily for SVG-based UIs; performance on other domains and more complex SVG graphics remains unvalidated.Gemini 2.5 Flash reaches 96.1% accuracy, leaving a 2.5% accuracy improvement for the method.Source: §6 · p. 10
  • The authors position structured function calling as a foundation for future skill-based abstractions, which still depend on function calling.The paper currently employs structured function calling for SVG editing.Source: §6 · p. 10

7 Conclusion

MM-SVGEdit combines visual grounding and modification in a two-stage system that supports both direct manipulation and natural-language interaction. The authors conclude that it offers a practical approach to UI structure understanding and editing.

7 Conclusion

  • The system uses a two-stage strategy—visual grounding followed by modification—with both stages supporting two interaction modalities.The conclusion describes the result as a high-precision, efficient, and controllable SVG editing system.Source: §7 · p. 10
  • The authors construct a UI-domain SVG dataset to enable systematic training and evaluation of LLMs.The conclusion identifies the dataset as one of the work's contributions.Source: §7 · p. 10
  • The authors present MM-SVGEdit as a practical solution for structural understanding and manipulation of user interfaces.The conclusion says the approach enhances editing accuracy and user controllability while reducing computational costs.Source: §7 · p. 10
Loading 2609.06116v1…