RULER: Instance-aware Rubric Rewards for SVG Generation

1 Introduction

• Open-ended instructions permit many valid renderings without absolute visual ground truth, while scalar metrics transfer poorly and can favor broken SVGs.These metrics also correlate weakly with human judgment within and across models.Source: §1 · p. 1
• Rubric scoring on 900 human-annotated SVG samples achieves Spearman’s ρ of 0.7929 and Goodman-Kruskal γ of 0.7574 against human judgments.The rubric-based evaluator correlates substantially better with human preference than conventional scalar metrics.Source: §1 · p. 2
• RULER generates six-item, instruction-specific rubrics across semantic, visual, and stylistic axes; a judge scores rendered rollouts item-by-item for GRPO rewards.The rubrics come from text alone, so the method requires neither paired SVG ground truth nor human preference labels.Source: §1 · p. 2
• 0.432/0.395 to 0.693/0.683 rubric scores on MMSVG-Illustration and MMSVG-Icon, surpassing dedicated SVG specialists and matching DeepSeek-V3.RULER also outperforms standard RL baselines and remains competitive on conventional metrics.Source: §1 · p. 2

2.1 SVG Code Generation

• Existing SVG reward paradigms vary in whether they require paired references, inspect rendered images, or provide multi-dimensional feedback; RULER combines reference-free, visually grounded, multi-dimensional, instance-conditioned criteria.Pixel metrics require paired references; code-length rules inspect code alone; embedding scores are coarse-grained and gameable under RL; universal rubrics omit instruction-specific correctness.Source: §2.1 · p. 3

2.2 Rubric-Based Evaluation and Rewards

• Earlier rubric-based RL applied fine-grained checklist rewards to text tasks, whereas RULER scores rendered SVGs against instance-aware rubrics generated without ground-truth SVGs.This provides case-specific supervision without restricting generation to a single reference SVG.Source: §2.2 · p. 3

3.1 Task Definition

• SVG generation is modeled as token-by-token policy sampling followed by deterministic rendering, making reward design—not the optimization machinery—the central challenge.The generated image is evaluated after the SVG sequence terminates.Source: §3.1 · pp. 3–4

3.2 Human Assessment of Existing Metrics

• ρ = 0.7929 for rubric scoring versus 0.6051 for Aesthetic and 0.5518 for CLIP; γ = 0.7574 versus 0.5465 and 0.5295, respectively.The metrics are sample-level Spearman correlation and pairwise Goodman-Kruskal agreement, respectively.Source: Figure 3 · p. 4, §3.2 · p. 4

3.3 Instance-Aware Rubric Generation

• RULER derives six instruction-specific criteria across semantic, visual, and stylistic axes, with items designed around observable intentions rather than exact pixel or path constraints.Each item is independently judgeable from the rendered image, and rubric items carry importance weights.Source: §3.3 · p. 5

3.4 Policy Optimization

• A judge VLM scores each rendered rollout against the rubric item by item, and weighted satisfactions form a normalized reward.This replaces an opaque scalar with a dense, multi-dimensional signal.Source: §3.4 · p. 5
• GRPO compares rewards among rollouts for the same instruction, forms group-normalized advantages, and updates the policy with a clipped surrogate objective.The process iteratively refines the policy to maximize satisfactions across semantic fidelity, visual quality, and rendering style.Source: §3.4 · p. 5

4.2 Main Results (RQ1)

• 0.693 on Illustration and 0.683 on Icon are RULER’s universal Rubric scores, versus SVG-specialist scores of 0.390 and 0.586, while matching DeepSeek-V3.Table 2 summarizes the main benchmark comparison.Source: Table 2 · p. 6, §4.2 · p. 6
• 53.3% against VectorFusion to 96.5% against JanusCoder were RULER’s non-tie win rates; it also remained competitive on CLIP, Aesthetic, and HPS.The blinded preference study covered 150 prompts; the universal Rubric score ranked RULER highest.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 Reward Design Analysis (RQ2)

• 0.660 on Illustration and 0.591 on Icon are Universal Rubric RL scores, improving over zero-shot but trailing RULER by 0.033 and 0.092.A query-agnostic checklist lacks prompt-specific notions of correctness.Source: §4.3 · p. 7
• 6.697 on Illustration and 6.210 on Icon are C+A+H RL Aesthetic scores, while Illustration CLIP falls from 0.244 to 0.196 and Icon Rubric from 0.395 to 0.262.On Icon, the policy produced overlapping strokes and repeated paths, averaging 6.3k tokens versus 0.3k for zero-shot.Source: §4.3 · p. 7
• 0.693 on Illustration and 0.683 on Icon are RULER’s Rubric scores, versus zero-shot scores of 0.432 and 0.395, the strongest overall performance among tested reward designs.The comparison fixes Qwen3-8B and GRPO while varying the reward signal.Source: §4.3 · p. 7, Table 4 · p. 8

4.4 Ablation Study (RQ3)

• 0.693 →0.580 is the largest Rubric-score drop when removing an axis; removing Rendering yields 0.599 and removing Semantic Fidelity yields 0.623.All three axes provide complementary training signals; Figure 4 reports rubric-design ablations across both benchmarks.Source: Figure 4 · p. 8, §4.4 · p. 8
• 0.536 is Rubric-S’s universal Rubric score; its stricter, structurally focused prompt triggered readable prompt-text shortcuts despite a text-hint penalty.The finding suggests rubric-prompt changes can reopen degenerate optimization channels.Source: §4.4 · p. 8

4.5 Robustness Analysis (RQ4)

• 0.561 ± 0.008 on Illustration and 0.559 ± 0.006 on Icon are RULER-4B Rubric scores; RULER-8B also exceeds Qwen3-32B in the reported setting.RULER-8B scores 0.692 ± 0.007 and 0.662 ± 0.016, versus Qwen3-32B’s 0.585 ± 0.005 and 0.566 ± 0.026.Source: §4.5 · pp. 8–9, Table 5 · p. 9

• Multi-axis rubric evaluation aligns closely with human judgments, with strong sample-level correlation and pairwise ranking agreement.The conclusion frames this as the empirical basis for RULER.Source: §5 · p. 9
• RULER turns instance-aware rubrics into dense rewards for SVG-generation policy optimization via GRPO.The approach extends rubric-based RL to open-ended visual code.Source: §5 · p. 9
• Across MMSVG benchmarks, RULER mitigates reward hacking, outperforms scalar-reward RL baselines and SVG specialists, and remains effective across base models and rubric generators.Source: §5 · p. 9

• RULER depends on an external rubric generator and VLM judge, so reward quality may inherit their biases, preferences, and failure modes.The judge may overvalue superficial cues or underweight subtle stylistic qualities, especially on prompts outside the training distribution.Source: Limitations · p. 10
• Rendering sampled SVGs and querying a judge VLM at each training step makes RL substantially more expensive than lightweight scalar or heuristic rewards.This cost may limit scaling to larger models, longer rollouts, or broader hyperparameter searches.Source: Limitations · p. 10
• The semantic, visual, and stylistic decomposition may not capture every valid artistic intent or domain-specific preference.Interactive, human-steerable, or domain-adaptive rubric design remains future work.Source: Limitations · p. 10

• The policy is trained with GRPO using VERL and the vLLM rollout engine, generally with Qwen3-8B.Thinking mode is disabled; prompt and response limits are 512 and 4096 tokens.Source: A Training Details · p. 12
• Training samples eight rollouts per prompt at temperature 1.0, with batch size 128 and learning rate 1 × 10−6.The setup uses PPO mini-batches of 64, micro-batches of 8 per GPU, entropy coefficient 0.001, and no KL regularization.Source: A Training Details · p. 12
• Experiments run on one node with eight H800 80 GB GPUs; rollout uses bfloat16, tensor parallelism of 8, and gradient checkpointing.The per-GPU token budget for log-probability computation is 65,536.Source: A Training Details · p. 12

B Dataset Construction

• Rubrics are generated for 20K Icon prompts and 12K Illustration prompts, then paired with their source prompts as RL reward specifications.Rubrics failing format or item-structure checks are discarded; rubrics whose ideal SVG scores below 0.9 are also removed.Source: B Dataset Construction · p. 12
• 30,737 filtered prompt–rubric pairs form the training set: 19,531 for MMSVG-Icon and 11,206 for MMSVG-Illustration.Training and test prompts are generated separately, but both splits remain within the benchmark distribution and are not treated as an out-of-distribution evaluation.Source: B Dataset Construction · p. 12

C Reward Computation and SVG Rendering

• The frozen Qwen3-VL-8B judge scores six rubric items independently on each rendered SVG, and their normalized weighted average supplies the reward.For each prompt, eight rollouts are rendered and scored; rewards are normalized within the group to form relative advantages for GRPO.Source: C Reward Computation and SVG Rendering · p. 12
• Standard RULER uses only its six instance-aware rubric items; the RubricS ablation instead uses a five-item rubric and a fixed text-hint penalty.The standard rubric discourages readable-text shortcuts by requiring visual communication of the prompt concept and penalizing shortcuts under rendering quality.Source: C Reward Computation and SVG Rendering · p. 13
• All SVGs are rendered with CairoSVG 2.9.0 at 512 × 512 on white before reward evaluation and metric computation.The same rendering pipeline is used for RULER and all baselines; metrics are aggregated over successfully rendered samples.Source: C Reward Computation and SVG Rendering · p. 13

• The two human studies address complementary questions: metric alignment with human judgments and direct comparison of RULER’s final outputs.Source: D Human Evaluation · p. 13
• One study evaluates how closely automated metrics align with human judgments of SVG quality.Source: D Human Evaluation · p. 13
• The other directly compares RULER’s final outputs with representative baselines.Source: D Human Evaluation · p. 13

• The study sampled 300 prompts evenly across MMSVG-Icon and MMSVG-Illustration, collecting outputs from three models for 900 rendered SVGs.Source: §D.1 · p. 13
• Ten annotators assigned each anonymized output a holistic 0–100 score, and three validators reviewed and rescored cases deemed unreasonable.Source: §D.1 · p. 13
• Human-alignment evaluation used GPT-5-mini with a shared universal rubric, separate from the Qwen3-VL-8B judge scoring prompt-specific rubrics during training.Source: §D.1 · p. 13
• 0.7929 Spearman ρ and 0.7574 Goodman–Kruskal γ: the universal Rubric score exceeded Aesthetic and CLIP against human scores on both measures.Source: §D.1 · p. 13

D.2 Final-Output Human Preference Study

• Human comparisons showed prompt and rendered candidates to annotators, hid model identities, randomized left–right order, and counted a render failure against the failing candidate.The non-tie win rate excludes ties from its denominator.Source: §D.2 · p. 13
• RULER achieved a non-tie win rate above 50% against all five evaluated baselines in blinded pairwise human comparisons.The study used 150 MMSVG-Bench prompts and compared RULER with Qwen3-8B, Qwen3-32B, VectorFusion, OmniSVG, and JanusCoder.Source: §D.2 · p. 13

E.1 Stability Across Inference Runs

• Across five inference runs, RULER scored 0.692±0.007 on MMSVG-Illustration and 0.662 ± 0.016 on MMSVG-Icon, outperforming Qwen3-8B and Qwen3-32B on Rubric score.The reported small run-to-run variation indicates stability across decoding seeds.Source: §E.1 · p. 14

E.2 Render Success Rates

• RULER successfully rendered 99.3% of Icon samples and all Illustration samples, leaving few excluded samples for its aggregate metrics.The main benchmark setting generated one output for each of 150 prompts per benchmark.Source: §E.2 · p. 14

E.3 Policy-Scale Analysis

• For Qwen3-4B, Rubric rises 0.372 to 0.561 on Illustration and 0.338 to 0.559 on Icon; RULER-4B approaches Qwen3-32B, while RULER-8B exceeds it on both.Results are averaged over five independently seeded inference runs.Source: §E.3 · p. 14

E.4 Rubric Generator Analysis

• Claude-Opus-4.6-generated rubrics score 0.662 vs. 0.646 for GPT-5.5-generated rubrics on Rubric; GPT-5.5 yields higher CLIP, while Claude leads Aesthetic and HPS.Source: §E.4 · p. 14
• Both generator variants exceed Qwen3-8B (0.394) and Qwen3-32B (0.566) on Rubric, with the Qwen3-8B policy held fixed across variants.Source: §E.4 · p. 14

E.5 Comparison with Iterative Diffusion Methods

• RULER generates SVG code in one autoregressive pass, whereas VectorFusion optimizes separately for each prompt, averaging 72.1 minutes and 31.4K tokens per sample.Source: §E.5 · p. 14
• 0.683 versus 0.510 on MMSVG-Icon and 0.693 versus 0.611 on MMSVG-Illustration: RULER achieves higher universal Rubric scores than diffusion-optimized methods.The comparison complements CLIP, Aesthetic, and HPS metrics, on which diffusion-optimized approaches can remain competitive or stronger.Source: §E.5 · pp. 14–15

F.1 Ground-Truth-Free Rubric Construction

• Rubrics use only the text instruction, not a paired reference SVG; an ideal SVG is an intermediate target, not a unique answer or an RL rollout reference.The rubric remains open-ended, using a shared six-item structure across Semantic Fidelity, Visual Quality, and Rendering Style, with weights (5, 5, 5, 4, 5, 5); item descriptions and scoring guides are instruction-specific.Source: §F.1 · p. 15

F.2 Rubric-S

• Rubric-S scores 0.536 on MMSVG-Illustration, below full RULER and all three single-axis ablations, and its policy increasingly inserts readable prompt-related text.Rubric-S emphasizes structural completeness, removes the default stylistic axis, and adds a fixed text-hint penalty item with weight −2; the result illustrates that stricter criteria need not yield a better reward signal.Source: §F.2 · p. 15

G Rubric Generation Cost

• $2,013.12 is the total cost of generating training rubrics, averaging $0.0655 per prompt; generation is an offline preprocessing step whose outputs are cached for RL.Claude-Opus-4.6 processes 60.18M input tokens and generates 68.49M output tokens.Source: G Rubric Generation Cost · p. 15

H.1 Rubric Generation Prompt

• The rubric-generation prompt first creates one plausible ideal SVG, then derives an instance-specific, open-ended rubric that permits different candidate shapes, layouts, orientations, colors, and implementations.Its requirements make the first two items prompt-specific semantic evidence and the remaining four visual-form, composition, rendering, and style criteria; the fixed structure separates observable dimensions.Source: §H.1 · p. 16

H.2 Rubric-based Judge Prompt

• The rubric-based judge scores each item independently, following only visible evidence and the rubric, with continuous decimal scores calibrated to the item’s scoring guide.Source: §H.2 · p. 17

H.3 Universal Rubric Judge Prompt

• The universal judge assigns one continuous overall satisfaction score based on six dimensions: concept, components, form, composition, rendering, and style.Its calibration reserves the highest scores for refined, strong outputs and discourages inflation for merely recognizable images.Source: §H.3 · p. 17

H.4 Rubric-S Generation Prompt

• The two-stage prompt generates an ideal SVG, then builds a self-contained rubric for judging candidate renderings without showing the ideal SVG to the judge.The ideal establishes a quality target while the rubric remains open-ended enough for different valid SVG implementations to score well.Source: §H.4 · p. 18
• The rubric has exactly five weighted items covering prompt readability, component legibility, subject shape complexity, internal-detail coverage, and visual layering.The fixed weights are 5 for each of the first four items and 4 for visual layering.Source: §H.4 · p. 18
• The five criteria must be orthogonal, independently judgeable, and grounded in separate observable dimensions rather than overlapping evidence.The prompt specifically separates overall readability from component legibility, shape complexity from internal detail, and internal detail from layering.Source: §H.4 · p. 18
• Each item requires concrete, item-specific score thresholds, reserving 1.0 for exceptional fulfillment and 0.0 for absence, contradiction, or unreadability.The calibration constrains crude recognizable drawings to at most 0.6 on most visual items and similarly caps primitive-only, detail-missing, or flat outputs on their respective criteria.Source: §H.4 · p. 18
• A fixed text-hint penalty scores readable prompt-related text as a semantic shortcut, with higher scores for clearer text and zero when no text-like hints appear.The guide also distinguishes partially readable text, faint text-like marks, and isolated strokes.Source: §H.4 · p. 19