PaperlayerView exact revision on arXiv

Source-linked AI summary

Rendering-Aware Reinforcement Learning for Vector Graphics Generation

Juan A. Rodriguez, Haotian Zhang, Abhay Puri, Aarash Feizi, Rishav Pramanik, Pascal Wichmann, Arnab Mondal, Mohammad Reza Samsami, Rabiul Awal, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli

arXiv:2505.20793v1cs.CVcs.AI

TL;DR

Autoregressive VLMs can generate SVG code, but training often lacks feedback from rendered outputs, and discrete decoding prevents direct use of differentiable rendering. RLRF uses reinforcement learning with composite rewards from rendered SVGs, and the paper reports improved SVG generation across tasks and model sizes compared with supervised fine-tuning.

  • Problem

    VLM-based SVG generation often lacks rendered-image feedback during training, while discrete autoregressive decoding prevents direct application of differentiable rendering.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    RLRF fine-tunes autoregressive VLMs with reinforcement learning, comparing rendered SVG roll-outs against input images using reconstruction, semantic-similarity, and code-efficiency rewards.

    Source: §1 · p. 2

  • Results

    RLRF improves SVG reconstruction, generation, and code efficiency across model sizes and reports gains over supervised fine-tuning.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    RLRF-equipped models produce semantically aligned, visually accurate outputs on benchmarks and SVG types not seen during training.

    Source: §6 · p. 10

    RLRF experiments may increasingly specialize models in SVG generation, potentially diminishing their general instruction-following capabilities.

    Source: §C.4 · p. 29

Abstract

from arXiv · show

Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-language models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF(Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.

1 Introduction

VLM-based SVG generation lacks feedback from rendered outputs, and discrete autoregressive decoding prevents direct use of differentiable rendering. RLRF uses rendered comparisons as reinforcement-learning rewards, combining reconstruction, semantic alignment, and code efficiency; the paper reports improvements across tasks and model sizes.

1 Introduction

  • The overview illustrates an out-of-distribution case without a ground-truth SVG where RLRF produces a meaningful graphic with gradient-rendered shadows while SVG-SFT fails.This is a qualitative example rather than a benchmark comparison.Source: Figure 1 · p. 2
  • Supervised SVG generation can lose consistency on long sequences and generalize poorly because training does not evaluate rendered outputs.Token-level losses support syntactic correctness but do not provide visual feedback, leaving models prone to hallucination, looping, or losing grounding on complex inputs.Source: §1 · p. 2
  • Discrete token sampling blocks gradients through autoregressive SVG generation, so differentiable rendering methods such as DiffVG cannot be applied directly.DiffVG instead backpropagates image-space losses to continuous primitive parameters.Source: §1 · p. 2
  • RLRF samples multiple SVG roll-outs, renders them, and compares their images with the input to provide evaluative feedback for reinforcement learning.The overview also depicts non-differentiable steps handled through RL.Source: §1 · p. 2, Figure 1 · p. 2
  • Its composite reward combines pixel-level reconstruction, semantic similarity, and SVG token-length efficiency.Reconstruction uses metrics such as L2 distance, while semantic similarity can use DreamSim or CLIP.Source: §1 · p. 2
  • RLRF is presented as the first online RL algorithm for inverse rendering code generation, using rendering rewards to optimize SVG generation.Source: §1 · p. 2
  • RLRF demonstrates state-of-the-art improvements in SVG generalization, fidelity, and code compactness across tasks and model sizes.The paper also reports substantial gains over supervised fine-tuning in accuracy, efficiency, and semantic coherence.Source: §1 · pp. 2–3

2 Related Work

Prior SVG-generation work spans image processing, latent-variable models, and language-model code generation, while VLMs and RL have also advanced across related tasks. The paper positions RLRF as adding rendered visual feedback to SVG generation and code-oriented RL.

SVG Generation

  • Traditional image-processing methods trace contours and cluster regions but produce verbose, semantically unstructured code; latent-variable models support interpolation and style transfer but use simplified SVG subsets.Source: §2 · p. 3
  • LLM-based methods capture full SVG syntax and explore structured generation, editing, and reasoning, but often lack visual validation and generate inefficient or inaccurate code.Source: §2 · p. 3

Vision-Language Models (VLMs)

  • VLMs and code-generating LLMs support inverse rendering across SVG, TikZ, and CAD, yet can hallucinate or generalize poorly.Source: §2 · p. 3

Reinforcement Learning Post-Training

  • Prior code-generation RL focuses mainly on execution correctness or execution-based refinement, whereas RLRF rewards SVG visual fidelity, semantic alignment, and code efficiency.CodeRL uses actor-critic optimization for functional correctness, while RLEF iteratively refines code using execution signals.Source: §2 · p. 3

3 Method

RLRF uses reinforcement learning to refine SVG generation by comparing rendered outputs with their inputs, rather than relying on differentiable rendering. Its two-stage training and composite rewards target visual fidelity, semantic alignment, and code efficiency.

3 Method

  • RLRF uses rendered SVG comparisons as evaluative feedback to train autoregressive VLMs, whose discrete token generation is not compatible with direct gradient propagation through rendering.The method renders sampled SVGs and evaluates them against the input using a composite reward.Source: §3 · pp. 3–4

3.1 Two-Stage Training

  • Training first adapts the VLM with supervised fine-tuning on Im2SVG, then post-trains it with reinforcement learning on renderings of its predictions.After SFT, the model can generate plausible SVG code but receives no feedback about rendered visual quality.Source: §3.1 · p. 4
  • RLRF treats the conditional model as a stochastic policy and uses GRPO to optimize rendered-output rewards without an external value baseline.GRPO centers rewards within a batch and uses PPO-style clipped ratios to keep successive policies close; the objective also includes KL regularization to the frozen SFT model.Source: §3.1 · p. 4

3.2 Rewards for Vector Graphics Rendering

  • The reward suite combines automated signals for rendered-image fidelity, semantic alignment, and SVG code efficiency using tunable weights.The framework evaluates rasterized images and underlying SVG code, using CairoSVG for rendering.Source: §3.2 · pp. 4–6
  • The pixel-level L2 reward compares normalized input and rendered images with clipping, while an edge-aware variant emphasizes structural alignment.The normalization is intended to reduce the influence of large uniform regions and exposure artifacts; the edge variant applies Canny detection, dilation, and blur.Source: §3.2 · p. 5
  • Semantic rewards use DreamSim, optionally on edge maps, for Im2SVG and CLIP text-image similarity for Text2SVG.The method also describes a VLM judge for Text2SVG generation quality.Source: §3.2 · p. 5
  • SVG Length Deviation penalizes excessive token length relative to the ground-truth SVG while allowing moderate variation.The reward applies a quadratic penalty when predicted length exceeds half the ground-truth length and is clipped to [−1, 1].Source: §3.2 · pp. 5–6
  • Sampling temperatures up to 1.2 improve exploration, but higher values produce unstable, diverged outputs; increasing rollout counts improves MSE, reward, and SVG length.The temperature ablation reports consistent improvement in Test MSE, Reward, and SVG Length measurements, while the rollout ablation reports training curves averaged over many roll-outs.Source: Figure 4 · p. 6, Figure 5 · p. 6

4 Experiments

The experiments post-train several VLMs for Im2SVG and also test text-conditioned generation, using varied training data and rollout settings. Evaluation spans multiple SVG benchmarks, baseline families, and image-fidelity, perceptual, efficiency, and text-alignment metrics.

4.1 Experimental Setup

  • Im2SVG RLRF post-training uses 20k high-entropy SVG-Stack samples and 64 roll-outs per image for 500 steps, covering 16k unique images.The SFT starting models were fine-tuned on 1.7M image-SVG pairs; RLRF training is also applied to StarVector-1B.Source: §4.1 · pp. 6–7
  • Text2SVG training uses Qwen3-8B with 16k Flickr30k and MM-Icons captions as inputs and no SVG supervision.Training uses 16 roll-outs per input and runs for 1,000 steps.Source: §4.1 · p. 7
  • Reward-function studies run for 800 steps, while sampling-temperature, KL-divergence, and rollout-count ablations run for 150 steps on Qwen2.5VL-3B.The model is chosen to reduce exploration cost while checking whether findings generalize to larger models.Source: §4.1 · p. 7

4.2 Evaluation

  • Evaluation compares image-processing, deep-learning, and language/vision-language baselines across Im2SVG and Text2SVG benchmarks using fidelity, perceptual, efficiency, and alignment metrics.Im2SVG metrics include MSE, SSIM, DINO Score, LPIPS, and code efficiency; Text2SVG uses CLIP similarity and an LLM-based judge.Source: §4.2 · p. 7

5 Results

RLRF improves SVG generation across image and text tasks, while ablations show that SVG-SFT provides a necessary foundation and reward design and exploration affect learning. The reported gains include out-of-distribution generalization, though Text2SVG outputs still have limitations.

5.1 Main Results

  • RLRF sets a new state-of-the-art on SVG-Stack-Hard with consistent gains across all Im2SVG metrics.The comparison covers reconstruction, semantic alignment, and code efficiency; LIVE has high sampling cost.Source: Table 1 · p. 8
  • RLRF significantly improves reconstruction accuracy and code efficiency for StarVector-1B and Qwen2.5VL 3B and 7B.Qualitative examples show more coherent, aligned SVGs than methods that lack rendering awareness.Source: §5.1 · p. 8
  • With 16k natural captions and no paired SVG supervision, RLRF enables Qwen3-8B to generate prompt-aligned Text2SVG outputs.The model still struggles with precise Bezier curves and often uses basic geometric primitives.Source: §5.1 · p. 8

5.2 Ablation Studies

  • Combining SVG-SFT followed by RLRF achieves the best performance; running RLRF directly on instruction-tuned checkpoints yields poor results.The instruction-tuned models lack SVG proficiency to select appropriate primitives and explore effectively.Source: Table 2 · p. 9, §5.2 · p. 9

5.1 Main Results

  • RLRF models improve on SVG-Emoji, SVG-Fonts, and SVG-Icons despite no training exposure to those benchmarks, producing reasonable but imperfect outputs.The tested shifts include unseen styles with shadows and layered effects; outputs are described as syntactically clean and editable.Source: §5.1 · p. 9

5.2 Ablation Studies

  • Temperature 1.0 encourages diverse roll-outs, and increasing roll-outs per input from 8 to 64 consistently improves performance.Removing the KL penalty avoids early saturation, improves reward progression, and eliminates reference-model queries without observed instability.Source: §5.2 · p. 9, Figure 6 · p. 9
  • Combining L2, L2-Canny, DreamSim, and a code-length penalty gives the best performance across reconstruction, semantic coherence, and compactness.L2-based rewards alone favor precise reconstruction but show lower DINO scores for semantic coherence.Source: §5.2 · p. 10

6 Conclusion

RLRF fine-tunes autoregressive VLMs with rendered roll-outs and composite rewards, following SVG-SFT. The paper reports more efficient, semantically aligned, visually accurate SVGs, including on unseen benchmarks and SVG types.

6 Conclusion

  • RLRF combines SVG-SFT with reinforcement learning on rendered roll-outs and rewards balancing pixel accuracy, semantic alignment, and code efficiency.The conclusion reports improved visual fidelity, code efficiency, and structural understanding.Source: §6 · p. 10
  • The authors propose that RLRF's core idea can extend beyond SVGs to HTML/CSS/JavaScript, LaTeX/TikZ, 3D rendering code, and CAD.They frame it as a general framework for structured, code-driven visual synthesis.Source: §6 · p. 10

Appendix; A Additional Experiments and Results

A.1 Im2SVG Results

  • On SVG-Stack-Hard examples, RLRF improves Qwen2.5VL-7B's geometric accuracy and semantic alignment, with outputs described as more faithful than larger baselines.The comparisons include SVG-SFT Qwen2.5VL-7B, Qwen2.5VL-70B, and Gemini 1.5 Pro.Source: §A.1 · p. 17

A.2 Text2SVG Results

RLRF approaches OmniSVG's Text2SVG performance despite using no paired SVGs from the target datasets. Qualitative examples show coherent, prompt-aligned outputs, while the method remains below OmniSVG and has limitations in aesthetics and model scale.

A.2 Text2SVG Results

  • RLRF approaches OmniSVG's Text2SVG performance despite no paired SVG training on the target datasets, but does not yet outperform it.MM-Illustrations is an out-of-distribution evaluation set; Flickr30k and MM-Icons captions are used in training, while their SVGs are not.Source: §A.2 · pp. 17–18
  • RLRF outputs are described as semantically aligned and visually coherent, including for prompts with multiple interacting objects and compositional cues.The cited examples contrast with larger baselines that omit key elements or misinterpret structural cues.Source: §A.2 · p. 18
  • The Text2SVG model demonstrates generalization without paired SVG supervision or image input, using a VLM alone to evaluate rendered outputs.The reward signal is provided solely by Qwen2.5VL-7B.Source: §A.2 · p. 18
  • Qwen3-3B did not benefit from RLRF because the model lacked the SVG proficiency needed to benefit from reinforcement learning.The outputs are also described as not yet fully aesthetic.Source: §A.2 · p. 18

A.3 Generalization across SVG Benchmarks

RLRF generalizes to unseen SVG domains, improving fidelity over SVG-SFT while shortening generated code on SVG-Emojis. Its advantages vary by dataset and task: SVG-Icons pixel metrics are an exception, and very complex inputs can still trigger generation loops.

A.3 Generalization across SVG Benchmarks

  • On SVG-Emojis, Qwen2.5VL-7B SVG-SFT’s MSE improves from 6.39 to 4.93 and DINO from 90.99 to 93.50, with average SVG length reduced by approximately 1,500 tokens.RLRF models were instruction-tuned on SVG-Stack, then refined using rendered rollouts without SVG supervision; SVG-Emojis and SVG-Fonts were excluded from training.Source: §A.3 · pp. 18–19
  • On SVG-Icons, RLRF improves DINO and LPIPS over SVG-SFT, but has slightly worse MSE and SSIM.The authors attribute this discrepancy partly to sparse line drawings, where small stroke misalignments can strongly affect pixel metrics.Source: §A.3 · p. 19
  • Qualitative examples show RLRF better preserving shape, style, overlaps, and symmetry than SVG-SFT on unseen domains, while other VLMs often miss details or semantics.Figure 10 reports more coherent and visually accurate RLRF outputs than SVG-SFT on SVG-Emoji; the text describes improved disk-icon occlusion and dress symmetry.Source: §A.3 · pp. 19–20, Figure 10 · p. 20
  • On very complex inputs, RLRF can produce diverging code and fall into loops; the authors suggest more diverse and complex training data as a mitigation.They particularly identify images requiring longer, more challenging SVG sequences.Source: §A.3 · p. 20
  • Image-processing methods can achieve low pixel error on simple glyphs, but score lower on perceptual metrics and produce much longer, less semantically structured SVGs.RLRF is described as balancing pixel fidelity, structural similarity, semantic alignment, and code compactness.Source: §A.3 · p. 20

A.4 Ablation Study: Impact of Rewards

The reward ablation finds the strongest overall results with the full combination of pixel, perceptual, edge, and length signals. Individual rewards have distinct trade-offs: L2 favors reconstruction, while DreamSim alone favors perceptual alignment over precise reconstruction.

A.4 Ablation Study: Impact of Rewards

  • DreamSim-based rewards alone saturate early and yield weaker reconstruction scores, while showing stronger perceptual alignment as indicated by DINO.The reported trade-off is semantic consistency rather than precise pixel-level accuracy.Source: §A.4 · p. 22
  • Adding Canny to L2 slightly improves early convergence, while the Length term contributes to smoother, more compact SVGs.Combining L2 and L2-Canny also accelerates convergence compared with either alone.Source: §A.4 · p. 22
  • The full reward combination achieves the best overall performance, balancing high DINO and SSIM with low MSE and LPIPS.The comparison uses the Qwen2.5VL-3B model after SVG-SFT and evaluates checkpoints on SVG-Stack-Hard.Source: §A.4 · pp. 20, 22, Table 5 · p. 23
  • L2 alone remains competitive for minimizing MSE but lacks the broader alignment capabilities of perceptual and structural signals.The ablation describes L2-based rewards as favoring precise reconstruction while producing lower DINO scores.Source: §A.4 · p. 23
  • Across training curves, combinations including DreamSim alongside other signals tend to outperform DreamSim alone, and all reward components together provide a stronger, more stable learning signal.The figure caption describes L2 alone as strong for reconstruction, particularly MSE, but the all-component combination as best overall.Source: §A.4 · pp. 22–23, Figure 13 · p. 24

A.5 Cases of Reward Hacking

The authors report several cases where models exploit reward signals without improving output quality. They describe the exploit mechanisms and mitigations, including rendering-resolution controls and adjustment of the length reward.

A.5 Cases of Reward Hacking

  • Reward hacking occurs when a model exploits the reward signal without actually improving output quality.The authors report observing several such instances.Source: §A.5 · p. 24
  • A small-viewBox exploit can shrink the rendered image and its comparison target, artificially raising image-based reward; rendering instead at the reference image size and aspect ratio addresses it.The original rendering resolution followed the predicted SVG’s viewBox.Source: §A.5 · p. 24
  • The length-deviation reward can drive SVGs toward collapse because it continues rewarding shorter outputs down to half the ground-truth length.The authors found a more stable approach in reducing the length-reward weight to 0.1 early in training, then gradually increasing it.Source: §A.5 · p. 24
  • In Text2SVG, models can place the prompt text directly in an SVG, inflating similarity scores, especially with CLIP-based rewards.The mitigation strips text elements or related prompt-encoding primitives before rendering and adds a Qwen2.5-72B-based evaluator to the final reward mix.Source: §A.5 · pp. 24–25

B.1 Metrics

For Text2SVG, standard image-text metrics did not correlate well with the visual characteristics of generated SVGs, so the authors introduce a VLM-judged accuracy score. The accompanying dataset sections describe SVG-Stack preparation and construction of a harder curated test set.

B.1 Metrics

  • CLIP Score and CLIP Aesthetics are used for general image-text alignment, but the authors found they do not correlate well with generated SVG characteristics for general-purpose models such as Qwen2.5VL.They introduce an Accurate score judged by Qwen2.5VL-70B, with a prompt tuned on a held-out validation set to align scores with human preferences.Source: §B.1 · p. 25

B.2 Datasets

  • SVG-Stack is scraped from GitHub and rasterized into paired image-SVG examples; preprocessing reduces its 2.1M samples to 1.7M for SVG-SFT.Cleaning rounds decimals, removes XML headers, and filters long URLs and embedded base64 images.Source: §B.2 · p. 25
  • SVG-Stack-Hard addresses noisy, overly simple, short, broken, or empty samples in the original evaluation set through filtering and manual curation.The authors retain SVGs of at least 500 tokens, cluster using DINO image features, and stratify-sample 600 examples.Source: §B.2 · p. 25
  • The implementation uses LLaMA-Factory for SFT and EasyR1 and VERL for GRPO-based reinforcement learning, with vLLM for rollout sampling.The authors note that optimized sampling is important for SVG generation’s long context sequences.Source: §B.2 · p. 25
  • Training uses bf16 AdamW with a frozen vision tower during SVG-SFT and RLRF.The stated settings include gradient norm 1.0, weight decay 1 × 10^-2, and no learning-rate warmup.Source: §B.2 · pp. 25–26

C.1 Multimodal Architecture

The paper uses a generalist vision-language model for SVG generation and refines it with rendering-based feedback. It also describes reward choices, training strategies, and limitations, while situating RLRF among autoregressive SVG and vision-language approaches.

C.1 Multimodal Architecture

  • Qwen2.5-VL combines an image encoder with a decoder-only language model, and fine-tuning addresses its lack of SVG-specific pretraining while retaining broader instruction-following abilities.The model is a generalist with broader multimodal training, unlike SVG-specialized StarVector.Source: §C.1 · p. 27

C.2 Reward Details

  • VLM judging provides a stronger Text2SVG reward signal than CLIP, whose rewards are weak because generated SVGs fall outside natural-image training distributions.The VLM judge assesses semantic accuracy, visual resemblance, and aesthetic quality.Source: §C.2 · p. 28

C.3 Implementation Strategies for Stable and Robust RLRF Training

  • A dynamic output-length cap based on the longest ground-truth SVG plus a threshold reduces early rollout overhead and encourages shorter, cleaner sequences.The strategy becomes less important as generation improves.Source: §C.3 · p. 29

C.4 Limitations

  • Current-model context limits and SVG specialization that may diminish instruction-following abilities, alongside rollout-generation bottlenecks that introduce GPU idle time, constrain RLRF.The authors identify both context length and GRPO rollout efficiency as limitations.Source: §C.4 · p. 29

D.1 SVG Generation

  • RLRF improves generalization over supervised fine-tuning by post-training with visual feedback, addressing autoregressive SVG models’ lack of rendered-output evaluation.The approach optimizes rendering quality with automatic reward signals despite non-differentiable rendering.Source: §D.1 · p. 30

D.2 Vision-Language Models (VLMs)

  • VLMs can produce valid code and plausible outputs yet still suffer hallucinations, limited long-range coherence, and poor generalization without rendered-output feedback.The paper says learning from rendered predictions improves precision and generalization in code-driven visual tasks.Source: §D.2 · p. 31

D.3 Reinforcement Learning Post-Training

Prior RL post-training work often targets functional correctness, while RLRF uses rendered-image feedback to optimize visual quality alongside semantic alignment and code efficiency.

D.3 Reinforcement Learning Post-Training

  • GRPO stabilizes training by normalizing rewards across batches, removing the need for a separate value network and reducing variance [Shao et al., 2024, Guo et al., 2025].Source: §D.3 · p. 31
  • Prior code-generation RL methods use execution or functional-correctness signals, including critic-predicted correctness in CodeRL [Le et al., 2022] and compiler results in PPOCoder [Shojaee et al., 2023].Source: §D.3 · p. 31
  • RLRF differs from RL post-training focused on functional correctness by jointly optimizing visual fidelity, semantic alignment, and code efficiency through rendering feedback.Source: §D.3 · p. 31
Loading 2505.20793v1…