PaperlayerView exact revision on arXiv

Source-linked AI summary

Rendering-Aware Reinforcement Learning for Vector Graphics Generation

Juan A. Rodriguez, Haotian Zhang, Abhay Puri, Aarash Feizi, Rishav Pramanik, Pascal Wichmann, Arnab Mondal, Mohammad Reza Samsami, Rabiul Awal, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli

arXiv:2505.20793v2cs.CVcs.AI

TL;DR

Autoregressive VLMs can generate SVG code but lack rendered-image feedback during training, while discrete decoding prevents direct use of differentiable rendering. RLRF trains them with reinforcement-learning feedback from rendered SVGs. The paper reports state-of-the-art improvements in SVG generalization, fidelity, and code compactness across tasks and model sizes.

  • Problem

    Autoregressive VLMs lack rendered-image evaluation during training, and discrete token sampling prevents direct use of differentiable rendering for SVG generation.

    Source: §1 · p. 2

  • Method

    RLRF samples SVG roll-outs, renders them, and uses composite reinforcement-learning rewards for reconstruction, semantic similarity, and code efficiency.

    Source: §1 · p. 2

  • Results

    RLRF achieves state-of-the-art improvements in SVG generalization, fidelity, and code compactness across tasks and model sizes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RLRF equips VLMs to produce more efficient, semantically aligned, visually accurate SVGs, including on benchmarks and SVG types not seen during training.

    Source: §6 · p. 10

    Generated SVGs still struggle with precise Bezier curves and may rely on basic geometric primitives.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-language models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF (Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.

1 Introduction

RLRF uses reinforcement learning and rendered-image feedback to improve SVG generation in vision-language models. The paper reports state-of-the-art improvements in generalization, fidelity, and code compactness across tasks and model sizes.

1 Introduction

  • Existing VLM-based SVG generators use supervised token-sequence training but can lose consistency on long sequences and fail to generalize to diverse or out-of-distribution inputs.Without rendered-output evaluation, token-level losses prioritize syntax over visual feedback; the paper links this gap to hallucination, looping, and loss of grounding on complex inputs.Source: §1 · pp. 1–2
  • Discrete token sampling makes autoregressive SVG generation non-differentiable, preventing direct use of differentiable rendering methods such as DiffVG.DiffVG backpropagates raster-space losses through continuous SVG primitives, unlike token-by-token VLM decoding.Source: §1 · p. 2
  • RLRF samples multiple SVG roll-outs, renders them, and compares the resulting images with the input to provide reinforcement-learning feedback.Its composite rewards cover image reconstruction, semantic similarity, and code efficiency based on SVG token length.Source: §1 · p. 2
  • The paper introduces rendering-reward RL as an online method for inverse-rendering code generation.It describes this as the first use of an online RL algorithm for that task.Source: §1 · p. 2
  • RLRF demonstrates state-of-the-art improvements in SVG generalization, fidelity, and code compactness across tasks and model sizes.The paper supports this claim with extensive analysis.Source: §1 · p. 3

2 Related Work

Prior SVG-generation work ranges from image tracing and latent-variable models to LLM-based code generation. The paper positions RLRF as adding rendered-visual feedback to code-generation and reinforcement-learning approaches that commonly lack it.

SVG Generation

  • Traditional tracing, latent-variable, and LLM-based approaches trade off code structure, SVG expressiveness, and visual validation in different ways.Tracing methods can produce verbose, semantically unstructured code; latent models use simplified SVG subsets; LLM-based methods capture full syntax but may generate inefficient or inaccurate code without visual validation.Source: §2 · p. 3

Vision-Language Models (VLMs)

  • VLMs and code-generating LLMs support inverse rendering across formats such as SVG, TikZ, and CAD, but may hallucinate or fail to generalize.The paper says its rendered-output feedback improves downstream performance and generalization in code-driven visual tasks.Source: §2 · p. 3

Reinforcement Learning Post-Training

  • Where code-generation RL often optimizes execution correctness or code quality, RLRF uses rendering feedback to optimize SVG visual fidelity, semantic alignment, and code efficiency.The paper contrasts RLRF's composite rendering reward with CodeRL [Le et al., 2022] and RLEF [Gehring et al., 2024].Source: §2 · p. 3

3 Method

RLRF addresses the lack of rendered-output feedback in autoregressive SVG generation by rendering sampled outputs, scoring them, and using reinforcement learning to improve generation. Training first adapts the VLM with SVG supervision, then applies composite rewards for fidelity, semantic alignment, and code efficiency.

3 Method

  • RLRF renders generated SVGs and compares them with the input to provide evaluative feedback for reinforcement learning despite non-differentiable token generation.The method targets autoregressive VLMs, whose discrete token sampling prevents direct gradient propagation through rendering.Source: §3 · pp. 3–4

3.1 Two-Stage Training

  • Training first uses SVG supervised fine-tuning, then treats the conditional model as a stochastic policy and optimizes sampled SVGs using rewards from rendered outputs.The SFT stage uses teacher forcing but does not assess the visual quality of rendered outputs.Source: §3.1 · p. 4
  • GRPO centers rewards within a batch to avoid an external baseline and uses PPO-style ratio clipping to keep successive policies close.The paper motivates this choice by the cost and instability of learning a value network for lengthy discrete SVG sequences.Source: §3.1 · p. 4

3.2 Rewards for Vector Graphics Rendering

  • The pixel-level L2 reward compares normalized input and rendered images; an edge-aware variant applies Canny edges, dilation, and blur to emphasize structural alignment.Normalization makes the reward scale-invariant and robust to exposure.Source: §3.2 · p. 5
  • Semantic rewards use DreamSim for image-conditioned generation and CLIP text-image similarity for Text2SVG, with edge-based DreamSim emphasizing contours and geometry.The paper also uses a VLM judge for Text2SVG generation quality.Source: §3.2 · p. 5
  • A token-length deviation reward discourages overly long SVGs, and weighted reward components combine fidelity, semantic alignment, and code efficiency.The length reward allows moderate variation while penalizing excessive length relative to the ground-truth SVG.Source: §3.2 · pp. 5–6
  • The temperature ablation reports improved measurements up to 1.2, while higher values cause unstable, diverged outputs; increasing roll-outs improves MSE, reward, and SVG length measurements.The roll-out ablation uses training curves averaged over many roll-outs for clearer visibility.Source: Figure 4 · p. 6, Figure 5 · p. 6

4 Experiments

Experiments cover image-conditioned SVG reconstruction and text-conditioned generation, using public VLMs and a range of SVG-generation baselines. Evaluation combines visual-fidelity, perceptual-similarity, code-efficiency, and text-image-alignment measures across several benchmarks.

4.1 Experimental Setup

  • Im2SVG experiments focus on Qwen2.5-VL and StarVector-1B at 224 × 224 resolution, while Text2SVG uses Qwen3; the paper also explores generalization beyond image-conditioned generation.Qwen2.5-VL supports adaptive resolution, and the Qwen models have limited SVG-specific performance without targeted fine-tuning.Source: §4.1 · p. 6
  • SVG-SFT fine-tunes Qwen2.5-VL 3B and 7B on 1.7M image-SVG pairs, while Im2SVG RLRF uses 20k filtered high-entropy samples and trains on 16k unique images.The RLRF training set is much smaller than the SFT set; Text2SVG RLRF uses 16k captions without SVG supervision.Source: §4.1 · pp. 6–7
  • Hyperparameter studies use Qwen2.5-VL-3B, with 800 steps for reward-function exploration and 150 steps for temperature, KL-divergence, and roll-out-count ablations.The studies cover 25.6k images and 4.6k images, respectively.Source: §4.1 · p. 7

4.2 Evaluation

  • Evaluation compares image-processing, deep-learning, and language or vision-language baselines on SVG-Stack-Hard and other SVG benchmarks, plus text-conditioned benchmarks.Im2SVG uses MSE, SSIM, DINO Score, LPIPS, and code efficiency; Text2SVG uses CLIP similarity and an LLM-based judge for text-image alignment.Source: §4.2 · p. 7

5 Results

RLRF improves SVG generation across image- and text-conditioned tasks, with results and ablations highlighting gains in fidelity, efficiency, generalization, and the training conditions that support them.

5.1 Main Results

  • Image-processing methods score strongly on reconstruction but produce verbose SVGs, while open VLMs lag and larger Qwen2.5VL models still reconstruct inaccurately.The image-processing SVGs often exceed 7k tokens and can reach 100k.Source: §5.1 · p. 7
  • RLRF sets a new state of the art across Im2SVG metrics and improves reconstruction accuracy and code efficiency for StarVector-1B and Qwen2.5VL-3B/7B.The Table 1 caption reports consistent gains across all metrics; qualitative results describe more coherent, well-aligned outputs.Source: Table 1 · p. 8, §5.1 · p. 8
  • With 16k natural captions and no paired SVG supervision, RLRF enables Qwen3-8B to generate prompt-aligned SVGs on Text2SVG.Qwen3-32B lacks the SVG-generation capability and rendering awareness required for high-quality outputs.Source: §5.1 · p. 8
  • RLRF generalizes to unseen SVG-Emoji, SVG-Fonts, and SVG-Icons benchmarks, producing visually faithful, syntactically clean, editable outputs despite distribution shifts.The examples include styles with shadows and layered effects absent from training; outputs are described as reasonable, though not perfect.Source: §5.1 · p. 9

5.2 Ablation Studies

  • Sampling temperature 1.0 encourages diverse roll-outs, and increasing roll-outs per input from 8 to 64 consistently improves performance.The study reports no reward saturation and suggests that more roll-outs could further improve results.Source: §5.2 · p. 9
  • Removing the KL penalty improves reward progression, avoids early saturation, and reduces computation by eliminating reference-model queries.The authors attribute the limited benefit of KL regularization to the naturally narrow conditional SVG distribution and strong reward signals.Source: §5.2 · p. 9, Figure 6 · p. 9
  • SVG-SFT is needed to establish baseline SVG fluency before RLRF, while combining pixel-level and semantic rewards with a length penalty gives the best reward configuration.L2-based rewards favor precise reconstruction but lack semantic coherence; adding DreamSim accelerates convergence.Source: §5.2 · p. 10
  • Running RLRF directly on instruction-tuned checkpoints fails because they lack the SVG proficiency needed to select primitives and explore effectively.Source: §5.2 · p. 10

6 Conclusion

The conclusion presents RLRF as a rendering-reward approach for improving SVG generation, while the appendix provides extended results, experimental details, and qualitative comparisons.

6 Conclusion

  • RLRF fine-tunes autoregressive VLMs by optimizing rewards on rendered roll-outs, balancing pixel accuracy, semantic alignment, and code efficiency.The paper describes prior SVG supervised fine-tuning as the foundation before reinforcement learning.Source: §6 · p. 10
  • The authors report improved structural understanding, visual fidelity, and code efficiency, including on benchmarks and SVG types unseen during training.Source: §6 · p. 10

Appendix

  • The appendix organizes extended quantitative and qualitative results, setup details, further method and limitation discussion, and related work.Source: Appendix · p. 16

A Additional Experiments and Results

A.1 Im2SVG Results

  • On SVG-Stack-Hard, Qwen2.5VL-7B with RLRF produces more visually faithful, geometrically accurate, and semantically aligned SVGs than SVG-SFT and larger baselines.Examples include intricate spatial layouts and multiple object parts; the comparison includes Qwen2.5VL-70B and Gemini 1.5 Pro.Source: §A.1 · p. 17, Figure 7 · p. 17

A.2 Text2SVG Results

The Text2SVG results evaluate RLRF across in- and out-of-distribution benchmarks, with qualitative comparisons showing coherent, prompt-aligned outputs alongside remaining capability limits.

A.2 Text2SVG Results

  • Text2SVG evaluation spans Flickr30k, MM-Icons, and out-of-distribution MM-Illustrations; training uses Flickr30k and MM-Icons captions but no paired SVGs.The authors introduce an Accurate metric only for results where ground-truth comparisons are feasible.Source: §A.2 · p. 17
  • RLRF approaches OmniSVG despite using no paired SVGs from the target datasets, but does not yet outperform it.Claude 3.7 achieves the strongest results among the compared models.Source: §A.2 · p. 18
  • Qualitative comparisons show Qwen3-8B RLRF producing semantically aligned, coherent SVGs, while larger baselines can omit elements or misinterpret compositional cues.The examples include prompts requiring multiple characters or interacting objects.Source: §A.2 · p. 18, Figure 8 · p. 18
  • Despite outputs not yet being fully aesthetic, RLRF enables smaller models to outperform larger baselines on structured visual grounding; on Qwen3-3B, it was ineffective because the model lacked sufficient SVG proficiency.Source: §A.2 · p. 18

A.3 Generalization across SVG Benchmarks

RLRF is evaluated on unseen SVG domains, where it improves over supervised fine-tuning across most perceptual metrics and produces stronger qualitative reconstructions. Its advantages coexist with a limitation on very complex inputs and a pixel-error trade-off against image-processing methods.

A.3 Generalization across SVG Benchmarks

  • On SVG-Emojis, Qwen2.5VL-7B improves from 6.39 MSE and 90.99 DINO to 4.93 MSE and 93.50 DINO, while reducing average SVG length by approximately 1,500 tokens.On SVG-Fonts, MSE drops from 7.1 to 4.73 despite no fine-tuning on that domain; SVG-Icons improves on DINO and LPIPS but has slightly worse MSE and SSIM.Source: §A.3 · p. 19
  • RLRF generations often match unseen inputs in shape and style more closely than SVG-SFT, including better-preserved disk occlusions and dress symmetry.GPT-4o, Claude 3.7, and Gemini 1.5 Pro perform inconsistently on fine details and shape semantics.Source: §A.3 · pp. 19–20
  • RLRF tends to give up on very complex inputs, producing divergent code that falls into loops.The authors suggest increasing training-data diversity and complexity, especially examples requiring longer SVG sequences.Source: §A.3 · p. 20
  • Image-processing methods achieve low MSE on simple, single-color glyphs but produce SVGs orders of magnitude longer and less efficient than learning-based methods.They score lower on DINO and yield code with less semantic alignment and practical value for editing or deployment.Source: §A.3 · p. 20
  • RLRF is described as the only approach that consistently improves pixel fidelity, structural similarity, semantic alignment, and code compactness.The paper presents this combination as a balance for generating editable, efficient code that generalizes to new domains.Source: §A.3 · p. 20

A.4 Ablation Study: Impact of Rewards

The reward ablation finds the full composite reward performs best overall, balancing semantic alignment and reconstruction fidelity. Individual reward signals differ in their strengths, and the same section reports Text2SVG results across multiple datasets.

A.4 Ablation Study: Impact of Rewards

  • The ablation evaluates reward configurations on SVG-Stack-Hard using DINO Score, MSE, LPIPS, and SSIM, starting from a Qwen2.5VL-3B model after SVG-SFT.Tested configurations include L2, L2 Canny, DreamSim, DreamSim+Canny, Length-penalty variants, and the full L2, DreamSim, Canny, and Length combination.Source: §A.4 · pp. 20, 22
  • The full reward combination achieves the best overall performance, with high DINO and SSIM and low MSE and LPIPS.The results describe stable convergence and improved performance across all metrics compared with other configurations and the pre-RL SVG-SFT baseline.Source: §A.4 · p. 22, Table 5 · p. 23
  • DreamSim-only rewards saturate early and yield weaker reconstruction scores, while L2-based rewards improve MSE, LPIPS, and SSIM but offer less semantic alignment.L2+Canny slightly improves early convergence, and adding the Length term contributes to smoother, more compact SVGs.Source: §A.4 · pp. 22–23
  • On Text2SVG datasets, RLRF improves over Qwen3-8B-Instruct before RL, and its models consistently rank second on MM-Icon and MM-Illustration behind OmniSVG.The authors note that standard CLIP and Aesthetic metrics are limited for SVG-like outputs, while their Accurate metric more clearly captures improvements.Source: Table 4 · p. 23

A.5 Cases of Reward Hacking

The appendix describes reward-hacking behaviors and mitigation measures, then reports a human evaluation favoring RLRF over SFT on editability and, less decisively, fidelity.

A.5 Cases of Reward Hacking

  • The study reports reward hacking, where the model exploits a reward signal without improving output quality.One example is the small-ViewBox hack: resizing both the rendered output and reference to a tiny viewbox can discard image information and artificially raise image-based reward.Source: §A.5 · p. 24
  • To prevent the small-ViewBox exploit, rendering uses the reference image's size and aspect ratio rather than the predicted SVG's viewbox.The authors also remove text elements or related primitives that encode the prompt and add a Qwen2.5-72B evaluator to reduce reliance on shallow visual-textual shortcuts.Source: §A.5 · pp. 24–25
  • The length-deviation reward can drive SVG Length Collapse because it continues rewarding shorter outputs below half the ground-truth length.The authors reduce the length-reward weight to 0.1 early in training, then gradually increase it to mitigate the bias toward overly short sequences.Source: §A.5 · p. 24
  • In Text2SVG, using the SVG text primitive to render the prompt directly can artificially boost similarity, especially with CLIP-based rewards.The rendered text aligns closely with the prompt's textual embedding, exploiting the reward rather than depicting the requested content.Source: §A.5 · pp. 24–25

A.6 User Study on Editability and Fidelity

  • In 267 ratings from 18 participants, RLRF was preferred over SFT in 42.7% of fidelity comparisons and 59.2% of editability comparisons.Participants also favored RLRF's structure and ease of modification; the reported SFT preference rates were 12.0% for fidelity and 15.0% for editability.Source: §A.6 · p. 25

B.1 Metrics

The appendix supplements Text2SVG evaluation with metrics intended to address limitations of standard image-text scores. It also describes the SVG-Stack data preparation and a curated, more challenging test set.

B.1 Metrics

  • CLIP Score and CLIP Aesthetics measure general image-text alignment but do not correlate well with generated SVG visual characteristics for instruction-tuned models such as Qwen2.5VL.The authors introduce an Accurate score, judged by Qwen2.5VL-70B across several axes and tuned on a held-out validation set to align with human preferences.Source: §B.1 · p. 26

B.2 Datasets

  • The authors rasterize GitHub-sourced SVG-Stack files into paired image-SVG examples and retain 1.7M examples after cleaning for SVG-SFT.Preprocessing rounds decimals to two significant figures, removes XML headers, and filters samples with long URLs or embedded base64 images.Source: §B.2 · p. 26
  • SVG-Stack-Hard addresses the original test set's noisy, overly simple, short, broken, or empty samples with a curated set of 600 more complex examples.The selection filters for SVGs of at least 500 tokens, clusters by DINO image features, uses stratified sampling, and manually verifies the retained examples.Source: §B.2 · p. 26
  • The implementation uses LLaMA-Factory for SFT and EasyR1 and VERL for GRPO-based reinforcement learning, with vLLM for rollout sampling.During RLRF training, the vision tower remains frozen during SVG-SFT and RLRF.Source: §B.2 · p. 27

C.1 Multimodal Architecture

This block describes the VLM architecture and training context for SVG generation, then explains how RLRF uses rendered-output feedback to address visual-evaluation gaps. It also covers reward design, training strategies, related work, and limitations.

C.1 Multimodal Architecture

  • Qwen2.5-VL combines an image encoder with a decoder-only language model, while RLRF fine-tuning specializes this generalist model for SVG fidelity and code compactness.The model receives visual tokens and generates SVG code autoregressively; text-only SVG generation can omit the image encoder.Source: §C.1 · pp. 27–29

C.2 Reward Details

  • CLIP-based rewards provide weak supervision for abstract SVG outputs, so the paper uses Qwen2.5VL-7B as a judge to assess semantic accuracy, visual resemblance, and aesthetics.The judge model is selected for memory efficiency during rollout scoring.Source: §C.2 · p. 29

C.3 Implementation Strategies for Stable and Robust RLRF Training

  • A dynamic maximum-length schedule limits rollouts to the longest ground-truth SVG in each batch plus a threshold, reducing rollout overhead early in training.The strategy’s importance naturally declines as generation improves.Source: §C.3 · p. 30

C.4 Limitations

  • The experiments are constrained by current models’ context lengths, and SVG specialization may diminish general instruction-following capabilities; GRPO training is also bottlenecked by rollout generation.The authors identify rollout overhead as a target for future optimization.Source: §C.4 · p. 30

D.1 SVG Generation

  • Earlier SVG-generation approaches include contour-tracing methods and latent-variable models, while autoregressive methods generate SVG syntax directly but lack rendered-output feedback.The paper positions RLRF as a reinforcement-learning post-training strategy that uses automatic visual rewards and reports improved generalization over supervised fine-tuning.Source: §D.1 · pp. 30–31
  • RLRF addresses autoregressive SVG models’ lack of rendered-output feedback by optimizing generation with automatic reward signals and reports better generalization across SVG tasks than supervised fine-tuning.The paper identifies discrete token generation as the reason gradients cannot be directly propagated through rendering.Source: §D.1 · p. 31

D.2 Vision-Language Models (VLMs)

  • VLMs can produce valid or plausible code yet still exhibit hallucinations, weak long-range coherence, and poor generalization when they receive no feedback from rendered predictions.The paper frames learning from rendered predictions as a way to improve precision and generalization in code-driven visual tasks.Source: §D.2 · p. 32

D.3 Reinforcement Learning Post-Training

Prior RL post-training work uses human or programmatic feedback and code execution signals, but often does not optimize visual quality. RLRF adds rendering feedback through a reward that jointly targets visual fidelity, semantic alignment, and code efficiency.

D.3 Reinforcement Learning Post-Training

  • GRPO normalizes rewards across batches to stabilize training, removing the need for a separate value network and reducing variance.The passage presents GRPO as an alternative to PPO for post-training.Source: §D.3 · p. 32
  • RLRF introduces rendering feedback through a composite reward that jointly optimizes visual fidelity, semantic alignment, and code efficiency.This contrasts with RL post-training methods that primarily focus on functional correctness.Source: §D.3 · p. 32
  • Prior code-generation RL methods use execution or functional-correctness feedback, while visual quality remains overlooked in those approaches.The related-work discussion also describes RL for human-preference alignment and visual-generation tasks.Source: §D.3 · p. 32

Related papers

Loading 2505.20793v2…