PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning from Intermediate Renders for Image-to-Code Generation

Omri Kaduri, Kate Feingold, Phillip Isola, Tali Dekel

arXiv:2609.34587v1cs.CVcs.AI

TL;DR

A single terminal reward for image-to-code generation cannot distinguish helpful from harmful token decisions. IR4RL uses changes between intermediate renders as localized token-level feedback alongside the final outcome reward. Across Image-to-SVG and Image-to-TikZ, it improves over SFT and outcome-only GRPO, yielding state-of-the-art open-source models.

  • Problem

    A single final reward cannot distinguish the individual contributions of helpful and harmful code decisions, leaving token-level credit assignment unresolved.

    Source: §1 · p. 2

  • Method

    IR4RL turns visual-similarity changes between consecutive intermediate renders into localized rewards for the tokens that produced them, combining these with the final-render outcome reward.

    Source: §1 · p. 2

  • Results

    Across Image-to-SVG and Image-to-TikZ, IR4RL improves over SFT and outcome-only RL with GRPO, achieving state-of-the-art results among open-source models while producing substantially shorter programs.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Intermediate renderings provide process-level supervision for image-to-code post-training, with highly granular render-progress reward accounting for most of the gains.

    Source: §6 · p. 9

    Intermediate prefixes must be meaningful executable states that can be evaluated against the target, a condition that does not hold for arbitrary image-to-code tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement learning is increasingly used to post-train vision-language models for image-to-code generation, such as generating SVG code from a reference image, by optimizing rewards computed from the final rendered output. However, relying on a single terminal reward provides sparse feedback that is poorly aligned with the contribution of individual tokens. A generated program may contain operations that accurately reproduce some parts of the target image alongside others that introduce errors, yet all tokens are trained from the same final outcome. We observe that many intermediate code prefixes are not only executable, but already produce meaningful partial renders that reflect progress toward the target. This property provides a natural source of denser supervision during generation. Based on this observation, we introduce IR4RL, an RL framework with a token-level render-progress reward that turns changes between intermediate renders into localized feedback for the generated sequence. We evaluate our approach on Image-to-SVG and Image-to-TikZ generation. Across both tasks, our method improves over supervised fine-tuning and standard GRPO, yielding new state-of-the-art open-source models. This shows that intermediate rendering provides a simple and effective source of process supervision for RL post-training of image-to-code models.

1 INTRODUCTION

IR4RL addresses the inability of terminal rewards to distinguish helpful from harmful code decisions by using intermediate renders for localized feedback. Across Image-to-SVG and Image-to-TikZ, it improves over SFT and outcome-only GRPO, with shorter programs and state-of-the-art results among open-source models.

1 INTRODUCTION

  • A single final reward cannot distinguish helpful from harmful decisions within a program, although rendering-based RL grounds generated code in its visual outcome.SFT learns from reference programs without observing how code errors translate into visual errors.Source: §1 · p. 2
  • IR4RL uses changes in visual similarity between consecutive renders as local rewards, propagates them to the tokens that produced them, and combines them with the final-render outcome reward.Intermediate prefixes can be rendered as partial programs, revealing how the reconstruction evolves toward the target.Source: §1 · p. 2
  • The approach applies where intermediate program states can be rendered and evaluated and generated elements persist as the program evolves; the paper studies SVG and TikZ.The two settings differ in representation syntax and rendering pipeline.Source: §1 · p. 2
  • Across Image-to-SVG and Image-to-TikZ, IR4RL improves over SFT and outcome-only GRPO, achieving state-of-the-art results among open-source models while producing substantially shorter programs.The analysis finds render-progress reward accounts for most of the improvement, while combining process and outcome feedback performs best.Source: §1 · p. 2

2 RELATED WORK

Prior image-to-code approaches include optimization-based reconstruction and learned program generation, while rendering-based RL has generally relied on sequence-level rewards. IR4RL uses intermediate renders during training to provide localized visual feedback without human annotations or a learned verifier.

Image-to-code tasks

  • Image-to-SVG systems include optimization-based methods that can be slow and produce complex geometry, and VLM-based models trained on image-code pairs; learned models also generate TikZ programs.The related-work overview also places image-to-code across vector graphics, scientific figures, webpages, and CAD.Source: §2 · p. 3

Fine-grained credit assignment

  • Fine-grained process supervision has used human labels, automatic stepwise supervision, learned estimates, execution signals, and segment-level corrections across reasoning, code, and multimodal tasks.The cited examples include process or execution feedback beyond image-to-code.Source: §2 · p. 3

Learning from rendering feedback

  • IR4RL uses directly evaluable intermediate program states to measure visual-alignment changes between successive renders, obtaining localized process rewards without human annotations or a learned verifier.Prior sequence-level rewards collapse visual progress into a single outcome-level signal; IR4RL uses intermediate renders during training, leaving inference unchanged.Source: §2 · p. 3

3 PRELIMINARY

The setup contrasts reference-code supervised fine-tuning with reinforcement learning that scores rendered visual similarity. GRPO samples programs for each image prompt and trains from group-relative outcome rewards, while intermediate renders reveal progress that a final outcome alone cannot localize.

3 PRELIMINARY

  • SFT matches a reference program, whereas rendering-based RL rewards visual similarity between the target and the program’s rendering without using ground-truth code.SFT may penalize visually accurate programs that differ from the reference code.Source: §3 · p. 3
  • GRPO samples a group of programs for each image prompt and scores them with outcome rewards; its group-relative statistics use the reward mean and standard deviation.The policy objective includes a regularizer toward a reference policy.Source: §3 · pp. 3–4
  • An outcome reward scores only the final program, so it cannot localize decisions that helped or hurt reconstruction; intermediate renders reveal visual-quality changes during generation.Rollouts can reach similar final quality through different trajectories, and later operations can reverse earlier progress.Source: Figure 2 · p. 4

4 METHOD

IR4RL localizes visual feedback by scoring changes between executable prefixes, propagating render-progress rewards to tokens, and retaining outcome supervision on the completed program.

4.1 RENDER-PROGRESS REWARD

  • Render-progress rewards measure the change in visual score between successive executable prefixes, rewarding improvements and penalizing degradations.The method places boundaries after completed drawing commands and closes incomplete syntax to render each prefix.Source: §4.1 · p. 4

4.2 PROPAGATING RENDER-PROGRESS REWARDS

  • Because segment rewards arrive only at render boundaries, future render-progress rewards are propagated backward through tokens with exponentially decaying weights.λ controls propagation distance: at λ = 0 rewards apply only at their render boundary, while larger values extend their influence to earlier tokens.Source: §4.2 · p. 5

4.3 COMBINING PROCESS AND OUTCOME REWARDS

  • The method combines process supervision for each segment’s marginal contribution with outcome supervision for final quality.A rollout can accumulate positive deltas yet end far from the target, such as by stopping early.Source: §4.3 · p. 5
  • The process term alone accounts for most of the improvement over outcome-only, while combining process and outcome signals performs best.Each token is weighted by both final outcome and its visual progress during generation.Source: §4.3 · p. 5

5 APPLICATIONS

IR4RL is evaluated on Image-to-SVG and Image-to-TikZ, settings where partial programs can be rendered and their visual progress measured. Across both tasks, it improves reconstruction over supervised and outcome-only RL baselines, while analysis examines reward composition, propagation, render frequency, and sampling.

5 APPLICATIONS

  • The approach is evaluated on SVG and TikZ, where partial programs can be rendered and newly generated elements persist as generation continues.These properties make changes between consecutive renders informative about the visual contribution of newly generated code.Source: §5 · p. 5

5.1 IMAGE-TO-SVG

  • Our method improves OmniSVG-4B across metrics, beats outcome-only RL and RAFT, surpasses task-specific models at 4B, and compares favorably with larger VLMs.The study also reports shorter SVG programs and preference for its outputs in human and VLM evaluations.Source: §5.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

5.2 IMAGE-TO-TIKZ

  • Across visual reconstruction metrics, our method improves DeTikZify-v2 and post-training baselines and produces shorter programs, with a small expected decline in code-similarity metrics.The reported qualitative comparisons show more faithful reconstruction of visual structure and content than supervised and outcome-based GRPO models.Source: Figure 4 · p. 6, Table 3 · p. 8, §5.2 · p. 8

5.3 ANALYSIS

  • Process-only reward recovers much of the gain over outcome-only RL, while combining process and outcome rewards performs best across metrics.The qualitative reward-composition comparison shows the combined objective yielding the most faithful reconstructions.Source: Figure 5 · p. 7, §5.3 · p. 8
  • Increasing the process-reward weight improves peak test reward around α = 10, with larger values providing no further gain.Figure 6a presents the process-weight comparison.Source: §5.3 · p. 8, Figure 6 · p. 9
  • Reward propagation substantially improves performance, with the best result at λ = 0.9; fully undiscounted propagation performs worse.The analysis suggests preserving locality in the process signal is beneficial.Source: §5.3 · p. 8, Figure 6 · p. 9
  • Performance consistently decreases as intermediate renders become less frequent, with the best result when rendering after every command (N = 1).The default formulation renders after each completed drawing command.Source: §5.3 · p. 8, Figure 6 · p. 9
  • Our model achieves the highest best-of-K reward, with the largest margin at small K; the gap narrows as K increases.The authors interpret this pattern as training increasing the probability of high-quality generations rather than requiring large sampling budgets to find them.Source: Figure 7 · p. 9, §5.3 · p. 9

6 CONCLUSIONS

IR4RL uses changes between consecutive renders to provide localized rewards while retaining a global outcome signal, improving results across Image-to-SVG and Image-to-TikZ. The conclusion also states the method’s scope boundary, training-time cost, and several deployment and future-work considerations.

6 CONCLUSIONS

  • Across Image-to-SVG and Image-to-TikZ, IR4RL improves over supervised fine-tuning and outcome-only RL, achieving state-of-the-art results among open-source models.Highly granular render-progress reward accounts for most of the gains.Source: §6 · p. 9
  • Intermediate prefixes must be meaningful executable states evaluable against the target, limiting the method’s natural fit beyond SVG and TikZ.Intermediate rendering adds training-time computation but leaves inference cost unchanged.Source: §6 · p. 9
  • The authors identify Lottie animations, HTML interfaces, and 3D scene programs as possible extensions to other incrementally rendered representations.They also frame intermediate execution as a source of process-level supervision for credit assignment beyond terminal rewards.Source: §6 · p. 9
  • The paper reports using generative AI for implementation, drafting, editing, literature identification, and reference formatting, with author review of AI-assisted work.The authors state that code was tested and verified and that AI-assisted text was checked against results and revised.Source: AI USE STATEMENT · p. 10
  • The work includes a Prolific human evaluation and notes that reproducing copyrighted visuals was neither targeted nor evaluated, leaving mitigation to downstream safeguards.The remaining experiments use publicly available datasets and models.Source: ETHICS STATEMENT · p. 10

A.1 TEST-TIME SCALING

The SVG test-time scaling analysis finds that IR4RL performs best across sampling budgets, with its largest advantages at small K. Prefix closure enables dense render feedback by turning completed SVG drawing-command prefixes into renderable programs.

A.1 TEST-TIME SCALING

  • Across SVG sampling budgets, IR4RL achieves the highest reward, with the largest advantage at small K across three evaluation sets and four temperatures.The gap to baselines narrows as K increases, consistent with high-quality generations receiving greater probability rather than requiring large sampling budgets to discover them.Source: Figure A.1 · p. 15, §A.1 · p. 15

A.2 PREFIX CLOSURE

  • For SVG, boundaries follow each completed drawing command; any path-segment prefix can be closed with a default black color, and boundaries account for roughly 15% of tokens.OmniSVG’s tokenizer omits document and path wrapper tags, so the closure operator wraps the decoded commands in standard container boilerplate.Source: §A.2 · pp. 15–16
  • Prefix closure produces renderable intermediate programs, with the required closure depending on task and tokenizer: OmniSVG omits the opening SVG tag, whereas DeTikZify generates its opening code.The appendix illustrates the task-specific process for Image-to-SVG and Image-to-TikZ.Source: Figure A.2 · p. 16

A.3 TRAINING PARAMETERS

The appendix specifies the SVG visual-similarity score and training recipe, then compares model sizes and records baseline evaluation settings. Its model-size analysis reports similar post-training results for 4B and 8B models, with the 4B choice requiring lower computational cost.

A.3 TRAINING PARAMETERS

  • SVG visual similarity S is a scale-invariant normalized L2 emphasizing structural and color discrepancies, with blank or malformed SVGs assigned −1.The score lies in [−1, 1].Source: §A.3 · p. 16
  • OmniSVG-4B is trained for 300 steps on one B200 GPU using 700 SVG-Stack samples; scaling to 10k samples brings no consistent reconstruction-quality gain.The 700-sample set wins on Illustrations, while 10k wins on Icons, with marginal differences in both cases; the smaller set is adopted.Source: §A.3 · pp. 16–17

A.4 IMPACT OF BASE MODEL SIZE

  • Despite a significant base-model margin, applying IR4RL to 4B and 8B models produces similar results, motivating the lower-compute 4B choice.The 8B model produces approximately 20% longer completions after post-training.Source: Table A.2 · p. 17, §A.4 · p. 17

A.5 BASELINE PARAMETERS

  • The appendix lists baseline decoding and optimization settings, including 500 iterations and 128 paths for DiffVG and 200 iterations and 16 paths for LIVE.It also specifies model sampling settings and RAFT’s sampling and fine-tuning procedure.Source: §A.5 · p. 17

A.6 METRICS

The evaluation appendix specifies image-metric conventions, a Prolific preference study, and an automatic Gemini 3 Flash judging procedure. It also includes prompts used for evaluation with closed-source API models.

A.6 METRICS

  • CLIP is computed image-versus-image, DINO uses its base checkpoint, LPIPS uses AlexNet, and Aesthetic scores the generated image alone.Token counts use OmniSVG’s tokenizer, with leaked prose and thinking traces included for API responses.Source: §A.6 · p. 17
  • The Prolific study compares each baseline with the model output on 50 MMSVGBench samples, collecting 1,500 judgments from 30 participants.Six participants rated each comparison in a randomized two-alternative forced-choice format.Source: §A.6 · p. 17, Figure A.3 · p. 18
  • Gemini 3 Flash serves as an automatic VLM judge, answering the same comparisons shown to human participants with A or B and a brief justification.The appendix also shows system prompts for closed-source API evaluation on Image-to-SVG and Image-to-TikZ.Source: §A.6 · p. 18, Figure A.4 · p. 18

B.1 EXTENDED REWARD ABLATION FOR IMAGE-TO-TIKZ

The TikZ reward ablation follows the Image-to-SVG trend, while prefix closure turns incomplete TikZ prefixes into compilable files.

B.1 EXTENDED REWARD ABLATION FOR IMAGE-TO-TIKZ

  • The Image-to-TikZ ablation shows the same trend as Image-to-SVG: process reward contributes more to full-method performance.The ablation was conducted at 30% of the training schedule.Source: §B.1 · p. 18, Table B.1 · p. 19

B.2 PREFIX CLOSURE

  • TikZ boundaries are placed after completed drawing statements, accounting for roughly 5% of tokens.A statement is a semicolon-terminated drawing command such as \draw or \fill.Source: §B.2 · p. 19
  • Prefix closure tracks open braces, options, and environments, then appends their closers from innermost outward to produce a compilable file.Source: §B.2 · p. 19

B.3 TRAINING PARAMETERS

This section specifies TikZ similarity scoring, DeTikZify-8B training, baseline decoding, and evaluation conventions.

B.3 TRAINING PARAMETERS

  • Self-Sim compares target and rendered patch features by Earth Mover’s Distance; blank or malformed renders score −1, and the score correlates with human judgment.The features come from DeTikZify-v2’s fine-tuned SigLIP vision encoder.Source: §B.3 · p. 19
  • DeTikZify-8B was trained for 900 steps on ∼25k DaTikZ-v3 samples using LoRA with a frozen vision encoder.Training used a 1e-5 learning rate, 128 rollouts per update, and a single B200 GPU for 6 days.Source: §B.3 · p. 19

B.4 BASELINE PARAMETERS

  • General-purpose (M)LLMs use provider APIs, while VinciCoder and DeTikZify-v2 use default sampling; post-training variants use greedy sampling.Source: §B.4 · p. 19

B.5 METRICS

  • TikZ generations are retried up to 10 times for compilation; TED uses TeX-tokenized edit distance, and token counts use DeTikZify-v2’s tokenizer.Source: §B.5 · p. 19
Loading 2609.34587v1…