Source-linked AI summary
Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs
Matthew Perlman, James Beetham, Niels Da Vitoria Lobo, Amrit Singh Bedi, Mubarak Shah
TL;DR
SVG generation remains difficult for general-purpose VLMs because it requires valid code, spatial reasoning, and precise visual translation. The paper evaluates an inference-time harness combining visual feedback, structured editing, and constrained decoding, finding improved compilation but persistent visual-quality and self-correction limitations.
Problem
General-purpose VLMs have limited evidence of effective SVG generation despite SVGs’ practical importance and the field’s focus on rasterized images.
Method
The paper evaluates a lightweight harness that combines rendering, visual feedback, structured editing, constrained decoding, and iterative refinement across VLM scales and SVG tasks.
Results
Constrained decoding substantially improves compilation rates for most pipelines but generally lowers visual quality, while iterative refinement has inconsistent effects and often reduces compilation rates.
Takeaways & Limitations
Inference-time methods show promise for adapting off-the-shelf VLMs to SVG generation, but validity and visual quality remain distinct objectives requiring joint treatment.
Takeaways & Limitations
Current VLMs struggle to translate rendered visual errors into precise vector-level corrections, limiting the effectiveness of iterative refinement.
Abstract
from arXiv · showhide
Scalable Vector Graphics (SVGs) power much of the modern visual ecosystem, yet state-of-the-art generative models focus almost entirely on rasterized images. We explore whether inference-time methods can unlock SVG generation capabilities in off-the-shelf vision-language models (VLMs). We systematically evaluate a constrained iterative refinement harness that combines visual feedback, structured editing, and constrained decoding to characterize the capabilities and limitations of current VLMs for SVG generation. Across multiple VLMs and generation settings, we find that constrained decoding improves compilation success rates, while iterative refinement reveals a deficit in visual reasoning and self-correction. Our results highlight both the promise and current limitations of using inference-time methods to adapt general-purpose VLMs for SVG generation.
1. Introduction
SVGs are practical because they are compact, resolution-independent, and editable, but general-purpose VLMs struggle to generate valid code with precise visual structure. This paper tests whether inference-time tools can adapt off-the-shelf VLMs for SVG generation.
- SVGs support web, interface, and illustration content through resolution independence, compactness, and direct editability.
- General-purpose VLMs must jointly produce valid code, reason about spatial relationships, and translate precise visual details into SVGs.
- The study evaluates a lightweight harness combining rendering, visual feedback, structured editing, and constrained generation across model scales and SVG tasks.
2. System Overview
The proposed harness wraps VLMs with constrained outputs, rendering, visual feedback, and structured editing for iterative SVG refinement. It supports different decoding constraints and refinement granularities across model types.
- Harness: The harness parses each constrained SVG edit, updates and renders the image, then returns code and visuals to the VLM for another iteration.
- Constrained Decoding: Closed-source models use JSON schemas, while open-source models use a CFG over a tractable subset of SVG elements and attributes.
- Refinement Variants: Single-turn generation provides a baseline without iterative refinement or visual feedback.
- Refinement Variants: Unstructured refinement regenerates the full SVG, whereas semi-structured refinement selects an action and produces a snippet for an element.
- Refinement Variants: Structured refinement outputs an action and element parameters that the harness converts into a complete SVG snippet.
- Refinement Variants: Semi-Structured and Structured refinement use Add, Update, Remove, and Exit actions, with up to 20 steps versus 10 for Unstructured refinement.All variants include free-form reasoning before code or actions in JSON.
3. Results
Across MMSVG-Bench, constrained decoding substantially improves compilation for most pipelines but generally reduces visual quality. Iterative refinement has inconsistent visual effects and can lower compilation rates, especially when models cannot connect visual errors to SVG code.
- Constrained decoding generally produced lower-quality images across visual metrics and generation settings.The authors hypothesize that the limited grammar steers generation toward unlikely and visually flawed SVG elements.
- Constrained decoding substantially improves compilation rates for most pipelines.
- Iterative refinement inconsistently affects visual quality and often lowers compilation rates by creating more opportunities for breakage.
4. Discussion
The discussion concludes that reliable SVG generation requires jointly addressing structural validity and visual quality. It identifies constrained decoding and repeated feedback as insufficient alone and points to several directions for improvement.
- Reliable SVG generation requires more than syntactic constraints or repeated visual feedback.
- Constrained decoding can improve validity while reducing visual quality, making structural validity and visual quality distinct objectives.
- Future work may explore richer action spaces, alternative visual feedback, stronger constrained decoding, and prompt meta-optimization.
- Mechanistic analysis of failures to incorporate visual feedback may inform improvements to general-purpose VLM spatial reasoning.