Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback

Guotao Liang; Zhangcheng Wang; Juncheng Hu; Haitao Zhou; Ziteng Xue; Jing Zhang; Dong Xu; Qian Yu

Source-linked AI summary

Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback

Guotao Liang, Zhangcheng Wang, Juncheng Hu, Haitao Zhou, Ziteng Xue, Jing Zhang, Dong Xu, Qian Yu

arXiv:2604.20730v3Apr 22, 2026cs.CV

TL;DR

Existing SVG generation methods use blind code synthesis that underuses visual information needed for partial-canvas and occlusion reasoning. The paper introduces Render-in-the-Loop with VSF training and RaV inference, achieving strong MMSVGBench performance and data efficiency, while leaving combined VSF–RL optimization for future work.

Problem
Blind open-loop SVG generation does not expose intermediate visual outcomes, underusing MLLM vision priors for partial-canvas and occlusion reasoning.
Method
Render-in-the-Loop renders cumulative intermediate canvases for visual conditioning, using VSF training and RaV inference to guide and filter incremental primitive generation.
Results
Render-in-the-Loop outperforms strong open-weight baselines on MMSVGBench while using 0.85M samples versus OmniSVG’s 2M and InternSVG’s 16M.
Takeaways & Limitations
The results support visual feedback as a data-efficient paradigm for Text-to-SVG and Image-to-SVG generation.
Takeaways & Limitations
Inference incurs a 1.50× latency overhead, uses fixed 224 × 224 intermediate canvases, and cannot undo, delete, or replace accepted primitives.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have shown promising capabilities in generating Scalable Vector Graphics (SVG) via direct code synthesis. However, existing paradigms typically adopt an open-loop "blind drawing" approach, where models generate symbolic code sequences without perceiving intermediate visual outcomes. This methodology severely underutilizes the powerful visual priors embedded in MLLMs vision encoders, treating SVG generation as a disjointed textual sequence modeling task rather than an integrated visuo-spatial one. Consequently, models struggle to reason about partial canvas states and implicit occlusion relationships, which are visually explicit but textually ambiguous. To bridge this gap, we propose Render-in-the-Loop, a novel generation paradigm that reformulates SVG synthesis as a step-wise, visual-context-aware process. By rendering intermediate code states into a cumulative canvas, the model explicitly observes the evolving visual context at each step, leveraging on-the-fly feedback to guide subsequent generation. However, we demonstrate that applying this visual loop naively to off-the-shelf models is suboptimal due to their inability to leverage incremental visual-code mappings. To address this, we first utilize fine-grained path decomposition to construct dense multi-step visual trajectories, and then introduce a Visual Self-Feedback (VSF) training strategy to condition the next primitive generation on intermediate visual states. Furthermore, a Render-and-Verify (RaV) inference mechanism is proposed to effectively filter degenerate and redundant primitives. Our framework, instantiated on a multimodal foundation model, outperforms strong open-weight baselines on the standard MMSVGBench. This result highlights the remarkable data efficiency and generalization capability of our Render-in-the-Loop paradigm for both Text-to-SVG and Image-to-SVG tasks.

1 Introduction

Render-in-the-Loop reframes SVG generation as a step-wise process grounded in rendered intermediate canvases rather than blind code-only prediction. Visual Self-Feedback training and Render-and-Verify inference address the difficulty of mapping incremental visual states to primitives while achieving strong performance with less data.

Motivation: Open-loop SVG generation produces symbolic code without perceiving intermediate visuals, underusing MLLM vision priors for visuo-spatial reasoning.This disconnect makes partial canvas states and occlusion relationships difficult to represent textually.
Paradigm Innovation: Render-in-the-Loop renders partial SVG code and feeds the cumulative canvas back as a visual prompt for the next generation step.The loop targets sequential layering, occlusion, and layer dependencies that are visually explicit but textually implicit.
Motivation: Naively adding visual context to off-the-shelf MLLMs yields negligible improvement or degradation because they cannot condition the next primitive on intermediate visual states.The limitation motivates specialized training rather than direct deployment of the visual loop.
Synergistic Training and Inference: Fine-grained path decomposition constructs dense visual trajectories for Visual Self-Feedback training, while Render-and-Verify filters redundant or degenerate inference steps.VSF aligns visual perception with incremental code generation, and RaV acts as an inference-time guardrail against repetition.
Paradigm Innovation: Render-in-the-Loop is complementary to semantic tokens, curriculum learning, and reinforcement learning, although combining VSF with RL remains future work.VSF supplies step-wise visual state conditioning, whereas RL optimizes policies using validity, alignment, or fidelity rewards.
Strong Performance and Data Efficiency: Training on 0.85M samples surpasses OmniSVG trained on 2M samples and InternSVG trained on 16M samples on MMSVGBench.The reported result positions explicit visual feedback as more important than raw data scaling for high-quality vector graphics synthesis.

2 Methodology

Render-in-the-Loop reformulates SVG generation as an interleaved, step-wise process in which the model generates code, observes cumulative rendered canvases, and continues drawing from the full history. Fine-grained path decomposition supplies dense visual trajectories for VSF training, while inference-time verification filters invalid steps and prevents stalled generation.

Step-wise Drawing Formulation: The framework represents SVG generation as a multimodal sequence that interleaves the prompt, code fragments, and rasterized cumulative canvases.At each step, the model produces a code segment and receives the rendered canvas formed by all preceding segments.
Step-wise Drawing Formulation: At step t, the model conditions the next code segment on the prompt and the complete preceding code-and-canvas history.Visual inputs are encoded as tokens so the model can inspect previous rendering results before generating the next stroke.
Visual Self-Feedback Training: VSF training optimizes output code fragments and the END token over constructed drawing sequences while masking prompts and inserted image tokens.The objective supports learning both how to draw from the current canvas and when to stop when the canvas aligns with the target.
Data Construction: Fine-grained decomposition splits lengthy paths into geometrically independent or coupled subpaths, analyzes their dependencies, and merges connected components to preserve rendering fidelity.This avoids semantic breakage from naive truncation while retaining visually coupled elements within shared path elements.
Data Construction: The preprocessing increases the average number of path elements from 4 to 6, creating denser intermediate states and an implicit curriculum of simpler geometric compositions.Each decomposed SVG is rendered cumulatively, with each intermediate image inserted before the next code segment.
Render-and-Verify Inference: Render-and-Verify rejects invalid generation steps and forces END after repeated failures exceed a predefined retry limit, preventing stalled decoding and noisy additions.The mechanism aims to ensure that accepted drawing actions make concrete visual contributions.

3 Experiments

Experiments evaluate Render-in-the-Loop across Text-to-SVG and Image-to-SVG, comparing it with strong baselines and isolating VSF, RaV, data efficiency, occlusion handling, and inference cost. The method performs strongly across tasks, while remaining vulnerable to repetitive generation, unusual semantic states, complex scenes, and added latency.

Experimental setup: The evaluation covers MMSVGBench, Text-to-SVG and Image-to-SVG tasks, multiple baseline categories, qualitative comparisons, ablations, and inference-cost analysis.The baselines include optimization-, LLM/VLM-, and RL-based methods, with evaluation on Icon and Illustration subsets.
Quantitative comparison: Our method achieves competitive performance on both Icon and Illustration subsets and shows superior reconstruction on the complex Image-to-SVG Illustration subset.The Image-to-SVG analysis highlights higher semantic-structure fidelity and lower LPIPS than open-loop OmniSVG.
Quantitative comparison: 127.64 vs. 128.80 FID and 0.293 vs. 0.291 CLIP Score: our method surpasses InternSVG on the Text-to-SVG Icon subset despite using substantially less training data.The comparison is against InternSVG trained on 16M samples, whereas our method uses 0.85M samples.
Ablation studies: Naive multi-turn prompting without VSF provides no consistent gain and often degrades performance, whereas VSF improves semantic grounding and reduces FID from 155.84 to 137.86.Without VSF, models omit requested structures or hallucinate incorrect ones; VSF conditions generation on the evolving canvas.
Ablation studies: RaV suppresses degenerate repetition by rejecting candidates with negligible pixel differences, helping produce more complete and concise SVGs.Without RaV, the model may repeatedly redraw the same contour and leave the SVG incomplete.
Efficiency and limitations: Training on 0.85M samples matches or exceeds baselines trained on 2M and 16M samples, while visual feedback adds a reported 1.50× latency overhead.The latency increase closely tracks a 1.46× increase in generated tokens; rendering cost is described as negligible.
Robustness: Observing the canvas helps the model resolve layer ordering and occlusion ambiguities that open-loop generation often mishandles.The reported mechanism supports back-to-front drawing and reduces hidden geometries that waste tokens.
Failure case analysis: Representative failures include repetitive loops, generic outputs for unusual semantic states, and coarse compositions that omit details in complex scenes.These cases indicate remaining difficulty with abstract states and richly compositional prompts.

4 Conclusion

Render-in-the-Loop reframes SVG generation as an incremental visual process that conditions the model on its own intermediate renderings. The approach combines Visual Self-Feedback training and Render-and-Verify inference, while retaining inference overhead and limits on correction operations.

Render-in-the-Loop reformulates SVG generation from an open-loop textual process into an integrated, context-aware visual sequence.
Visual Self-Feedback training and Render-and-Verify inference support competitive capabilities and data efficiency on standard benchmarks.
1.50× latency overhead remains a limitation because intermediate SVG renderings are repeatedly inserted into the context window.
The 224 × 224 intermediate canvases leave the trade-off between finer visual detail and computational cost unexplored.
The additive formulation cannot delete, replace, or modify accepted primitives, so early errors can only be visually compensated by later additions.

Overview

The supplementary material provides additional analyses, implementation details, baseline descriptions, benchmark information, and related-work context for Render-in-the-Loop. It also includes a comparison with LIVE and DiffVG focused on stroke-stacking behavior.

Additional Experiments and Analyses covers RaV threshold sensitivity and supplementary qualitative results.
Implementation Details explains RaV filtering, model system prompts, and the visual self-feedback training format.
The supplementary material expands descriptions of the baseline models used in the comparative study.
The comparison with LIVE and DiffVG gives a more granular visual critique of stroke stacking in optimization-based methods versus structured generation.
MMSVGBench documentation describes the multimodal benchmark and its evaluation metrics.
The related-work section reviews optimization-based, autoregressive, and feedback-driven SVG generation and positions Render-in-the-Loop among them.

A Additional Experiments and Analyses

The supplementary analysis examines whether RaV remains stable across threshold settings and provides additional qualitative results.

The supplementary section covers RaV threshold robustness and additional qualitative results.

A.1 Robustness of RaV Thresholds

RaV remains stable across loose, default, and strict threshold configurations, supporting its use as a lightweight degeneracy filter rather than a brittle tuning-dependent module.

CLIP varies within 0.001 and HPS within 0.003 across Loose, Default, and Strict RaV configurations.
RaV filters visually non-contributive or near-duplicate fragments rather than optimizing semantic quality.
The stable results characterize RaV as a lightweight degeneracy filter rather than a brittle, benchmark-specific module requiring careful tuning.

A.2 Additional Qualitative Results

Additional results show accurate attribute binding and diverse, topologically clean SVG generation, while RaV performance remains stable across threshold settings.

The model accurately binds specific attributes, including colors and counts, to corresponding object parts.
Generated SVGs span diverse subjects while maintaining semantically coherent outputs and topologically clean path structures.
RaV performance is stable across loose-to-strict threshold configurations, indicating robustness as a degeneracy filter.

B Implementation Details

The implementation incrementally proposes SVG fragments, renders cumulative canvases, and verifies candidates using visual-difference and repetition checks before accepting them.

At each inference step, the model generates a candidate SVG fragment and renders a hypothetical future canvas before acceptance.
Cumulative rendering applies the candidate to previously accepted primitives, with R denoting SVG rasterization and C1:t−1 the accepted history.
Visual-difference verification rejects fragments whose rendered canvas changes negligibly, including redundant, occluded, or off-canvas elements.
Repetition checks reject structurally similar fragments, while adaptive resampling increases sampling variation until a candidate passes verification.
If no valid fragment is produced within Kmax retries, the system emits the END token and terminates drawing.
Intermediate canvases use 224×224 rasterization through CairoSVG, with thresholds ϵ = 0.001 and τsim = 0.98.
System prompts instruct the MLLM to follow iterative drawing and use rendered visual feedback for Text-to-SVG and Image-to-SVG tasks.

C More Details of the Baselines

The baselines include optimization-based, autoregressive multimodal, and reinforcement-learning approaches, differing in representation, scalability, fidelity, and computational cost.

Optimization-based Methods: Optimization-based methods use diffusion distillation or differentiable rendering to optimize vector graphics, often requiring computationally intensive iterative procedures.
Autoregressive LLM/VLM-based Methods: Autoregressive methods predict SVG primitives or tokenized commands from multimodal inputs, but some target simple icons or face context-length limitations.
Autoregressive LLM/VLM-based Methods: InternSVG uses large multimodal transformers trained on approximately 16M SVG samples, emphasizing scalability with large training corpora.
Reinforcement Learning-based Methods: SVGen applies GRPO-based reinforcement learning to optimize SVG generation policies against evaluation metrics.
The experiments use official baseline implementations with hyperparameters recommended in the original papers.

D Comparison with LIVE and DiffVG

Compared with LIVE and DiffVG, Render-in-the-Loop emphasizes interpretable structure and logical layering rather than fitting raster pixels through dense path stacking.

LIVE and DiffVG can achieve high visual fidelity through gradient descent, but their SVGs often lack human-interpretable structure and logical layering.
These methods may generate many redundant paths and overlapping layers, producing bloated SVG files that are difficult to edit, animate, or reuse.

E MMSVGBench

MMSVGBench evaluates multimodal SVG generation across text-to-SVG and image-to-SVG tasks using diverse perceptual metrics. Its samples are generated from scratch to avoid training-data leakage.

Benchmark composition: MMSVGBench contains 600 synthetic samples split evenly between text-to-SVG and image-to-SVG evaluation.Each task has 300 samples spanning icon-level and illustration-level complexity.
Benchmark composition: All benchmark prompts and images are generated from scratch, ensuring samples are unseen during model training.
Evaluation metrics: Text-to-SVG evaluation reports FID, CLIP score, Aesthetic score, and HPS, while image-to-SVG uses DINO similarity, SSIM, LPIPS, and MSE.
Baseline behavior: Optimization-based baselines can produce densely overlapping Bézier curves and redundant layers, yielding uneditable and semantically structureless vector representations.