PaperlayerView exact revision on arXiv

Source-linked AI summary

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein, Afsana Mimi, Marcella Cornia, Lorenzo Baraldi, Gerard de Melo

arXiv:2609.03806v1cs.AIcs.CV

TL;DR

SVG evaluation lacks domain-specific metrics that reliably reflect human judgments of semantic fidelity. SVG-Score combines human-rated Semantic Alignment data with adapted CLIP scorers and a trained VLM judge, and its evaluators agree better with humans than existing metrics while supporting generator benchmarking. Its benchmark rankings, however, rely on automatic evaluators rather than human annotations on the benchmark itself.

  • Problem

    Natural-image metrics such as CLIPScore align only weakly with human judgments of semantic fidelity for structured SVGs, while off-the-shelf VLM judges respond unevenly to SVG semantic errors.

  • Method

    SVG-Score builds a human-rated Semantic Alignment dataset and trains SVG-adapted, preference-aligned CLIP scorers alongside a supervised- and reward-trained VLM judge.

  • Results

    The proposed evaluators agree substantially better with human judgments than existing metrics and are used to benchmark proprietary, open-source, and optimization-based SVG generators.

  • Takeaways & Limitations

    SVG-Score provides a human-grounded framework for evaluating semantic alignment in text-to-SVG generation across generators and semantic complexity levels.

  • Takeaways & Limitations

    The evaluators assess Semantic Alignment only, not vector properties such as editability, path efficiency, layer organization, or code quality.

Abstract

from arXiv · show

Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \textbf{\ours}, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for \textit{Semantic Alignment}, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.

1. Introduction

SVG evaluation has lagged behind generation because natural-image metrics align weakly with human judgments of vector-graphic semantics. SVG-Score addresses this gap with human-rated alignment data and complementary evaluators that better capture semantic fidelity.

  • SVGs’ abstraction, distinctive color distributions, and sparse parametric structure create a domain shift for CLIP-based evaluation.
  • Controlled perturbations show that CLIP barely responds to wrong counts or spatial relations, while off-the-shelf VLM sensitivity varies across error types and SVG styles.
  • The framework combines SVG-adapted, preference-aligned CLIP scorers with a VLM judge trained using supervised fine-tuning and reward-shaped GRPO.
  • The proposed evaluators agree substantially better with human judgments than existing metrics and benchmark SVG generators across increasing semantic complexity.
  • SVG-Score introduces more than 12K human Semantic Alignment ratings covering 8,671 SVGs and 1,858 captions.

2. Related Work

Prior SVG research emphasizes generation and large-scale data resources, while evaluation commonly borrows protocols from raster imagery. SVG-Score instead trains evaluators directly on human ratings of text-to-SVG semantic alignment.

  • SVG Generation: Text-to-SVG generators either vectorize raster outputs, optimize paths through differentiable rasterization, or decode vector primitives directly.
  • SVG Datasets and Benchmarks: Existing SVG datasets often use automatically generated or weakly aligned captions, favoring pre-training over fine-grained semantic evaluation.
  • Automatic Judges for Visual Generation: Off-the-shelf VLM judges accept long prompts but remain insufficiently examined for structural precision and geometric fidelity in SVGs.
  • SVG-Score trains SVG evaluators directly on human ratings of text-to-SVG semantic alignment rather than transferring metrics from natural images.

3. Limitations of Current SVG Evaluation

Current SVG evaluators are tested with controlled semantic perturbations because standard CLIP and VLM judges may miss fine-grained compositional errors. The analysis measures score changes between corrupted and original caption-image pairs.

  • The perturbation study targets color, quantity, and spatial or relational inconsistencies while separately analyzing color and black-and-white SVGs.
  • Δ Score = Score_perturbed − Score_original; negative values indicate sensitivity to corrupted caption-image correspondence.
  • CLIP scores change little for spatial and count perturbations, respond somewhat more to color changes, and drop more for crude geometric proxies or uniform backgrounds.
  • Zero-shot Qwen3-VL-8B is generally more sensitive than CLIP, especially on color SVGs, but responds weakly to spatial and quantitative changes in black-and-white graphics.

4. Dataset Construction

The paper builds two disjoint resources for SVG evaluation: a human-rated Semantic Alignment dataset and an independent caption benchmark spanning increasing semantic complexity.

  • Human-Rated Semantic Alignment Dataset: The human-rated resource contains 8,671 SVGs, 1,858 captions, and 12,957 valid caption-SVG rating instances.Each instance receives an ordinal Semantic Alignment score from 1 to 5, from unrelated to semantically faithful.
  • Human-Rated Semantic Alignment Dataset: Ratings are collected by pairing each sampled caption with its original SVG and six retrieved SVGs spanning different alignment levels.The retrieved candidates are selected using the original SVG as a query under a loose similarity criterion.
  • Dataset Split: The train-test split is disjoint in both captions and SVGs, reserving 2,374 test instances for evaluator-human agreement.The training split has 10,583 instances, while the test split has 2,374 instances, 2,359 SVGs, and 339 captions.
  • Independent Generator Benchmark: The independent benchmark contains 1,616 human-validated captions divided into simple, medium, and compositional complexity levels.It is disjoint from the human-rated dataset and is not used to train the evaluators.

5. Model Training

SVG-Score trains two complementary human-aligned evaluators: preference-aligned CLIP models adapted to SVGs and a Qwen3-VL judge trained with supervised fine-tuning and reward-shaped GRPO.

  • Evaluator Overview: The framework combines an efficient SVG-adapted CLIP scorer with a VLM judge trained to reproduce human assessments.The two evaluators target complementary evaluation modes for text-to-SVG alignment.
  • SVG-Adapted CLIP: Three CLIP backbones are adapted on 3.2M SVG image-text pairs from StarVector and OmniSVG.The variants are ViT-B/32, ViT-L/14, and ViT-H/14, using non-overlapping examples between the sources.
  • Human-Preference Alignment: Human preference alignment trains each SVG-adapted CLIP model to rank higher-rated SVGs above lower-rated alternatives for the same caption.The base SVG-adapted CLIP weights are frozen while a pairwise objective is optimized.
  • Human-Aligned Qwen3-VL-8B Judge: The Qwen3-VL-8B judge first learns structured rationales and 1–5 scores through supervised fine-tuning, then receives ordinal and intra-caption ranking rewards through GRPO.The output format includes a short rationale and a predicted Semantic Alignment Score.

6. Experimental Results

Experiments show that SVG adaptation and preference alignment improve CLIP agreement with humans, while the trained VLM judge performs best among the reported evaluators and supports generator benchmarking.

  • Evaluation Protocol: Evaluator agreement is measured with Spearman’s ρ, Pearson’s r, Kendall’s τ, MAE, and pairwise ranking accuracy on held-out human ratings.Higher is better for the correlation and ranking metrics, while lower is better for MAE.
  • CLIP-Based Evaluators: SVG-domain adaptation improves agreement for every CLIP backbone, and preference alignment adds a further gain.On ViT-B/32, Spearman’s ρ rises from 42.90 to 54.26 after adaptation and to 59.31 after alignment.
  • CLIP-Based Evaluators: Our ViT-H/14 scorer reaches 63.18 ρ and 80.64 PA, outperforming HPSv2 by 7.95 and 4.98 points at identical parameter count.ImageReward nevertheless has the best MAE, while ViT-L/14 has higher Pearson correlation than ViT-H/14.
  • Ablation Study: The combined supervised-fine-tuning and GRPO model reaches 72.95 ρ and 0.69 MAE, whereas either stage alone improves only selected metrics.Adding the intra-caption ranking reward further raises ρ to 74.85 and τ to 65.08 with essentially unchanged MAE.
  • Perturbation Sensitivity: Adaptation increases CLIP sensitivity mainly to reduced-image perturbations, while the VLM judge improves across all five perturbation types by 1.4–4.9 points.The VLM judge exceeds CLIP’s gain on every caption perturbation category.
  • Generator Benchmark: On the independent benchmark, commercial systems perform best overall, with Claude Sonnet 5 ranking first under six of seven evaluators.HiVG leads the open-source group under the SVG-CLIP variants and the human-aligned VLM judge.

7. Conclusion

SVG-Score addresses weak sensitivity to fine-grained semantic errors with a human-rated Semantic Alignment dataset and evaluators aligned to human judgments.

  • Evaluator design: The framework combines SVG-adapted CLIP scorers with a human-aligned Qwen3-VL-8B judge.These evaluators are trained using the human-scored Semantic Alignment data.
  • Conclusion: The resulting evaluators agree substantially better with human judgments and respond more sensitively to semantic perturbations.This addresses the poor capture of fine-grained semantic errors by existing metrics.
  • Human-aligned evaluation: SVG-Score evaluates Semantic Alignment: how faithfully a rendered SVG reflects its caption.Human annotation uses a holistic 1–5 score rather than separate attribute scores.

A.2. Composition of the Datasets

The paper uses separate corpora for CLIP adaptation, human Semantic Alignment supervision, and independent generator benchmarking, with controlled construction and leakage safeguards.

  • CLIP adaptation corpus: The CLIP adaptation corpus contains approximately 3.2M SVG-text pairs from StarVector and OmniSVG, recaptioned with Qwen3-VL-8B.The corpus is used only for SVG-domain adaptation and contains no human Semantic Alignment supervision.
  • Generator benchmark: The independent benchmark contains 1,616 captions across three increasing levels of compositional complexity.It is never used to train the evaluators.
  • Data separation: Benchmark construction and human-supervision data use separate sources and leakage-control procedures.The benchmark evaluates generators on fixed prompts without providing source images to generators or judges.

B. Training and Inference

Training and inference combine deterministic SVG rendering, SVG-domain and preference adaptation for CLIP, and supervised plus reward-shaped training for the VLM evaluator.

  • Inference and rendering: All SVG metrics rasterize SVGs with CairoSVG at 448 × 448 pixels and assign invalid outputs score 0 for CLIP, HPSv2, and Aesthetic v2.VLM judges receive the rendered RGB image together with a caption-faithfulness prompt.
  • VLM training: The VLM evaluator is trained with supervised fine-tuning and GRPO using ordinal score proximity and intra-caption ranking rewards.The ranking term compares SVGs sharing the same caption, while the ordinal term rewards predictions closer to human scores.
  • CLIP training: CLIP training uses separate contrastive SVG-domain adaptation and HPS-style pairwise preference fine-tuning stages.The corresponding settings are reported for the CLIP adaptation and preference stages.
  • Reward design: The ordinal reward gives partial credit based on distance from the human score and adds a bonus for well-formed structured output.Invalid or non-parsable predictions receive a reward of −2.0.
  • Evaluation prompt: The Semantic Alignment prompt asks the VLM to assess object presence, attributes, spatial relations, counts, and overall semantic match on a 1–5 scale.The rendered SVG is provided alongside the associated caption.
  • Generation prompts: Commercial systems use a common SVG-generation prompt, while specialized open-source models use their official inference pipelines and default prompting procedures.The common prompt requests only SVG code with a 200-by-200 viewBox, height, and width.

C. Benchmark by Difficulty

Difficulty-stratified benchmarking shows that the trained judge provides the most stable and discriminative rankings, especially when captions describe interacting objects.

  • Ranking stability: Spearman ρ = 0.96 between Hard and full-set orderings for the trained judge, versus 0.94 for zero-shot Qwen and 0.87–0.93 for CLIP evaluators.The benchmark contains 1,000 Easy, 500 Medium, and 116 Hard prompts.
  • Generator separation: On Hard captions, the trained judge is the only evaluator that clearly separates the three best generators.The best-to-third-best distance grows from 0.42 to 2.07 points under the judge, compared with 1.61 to 2.87 under SVG-CLIP-B.
  • Difficulty effects: Relative to full-set scores, GPT-5 nano loses 42%, Gemini 3.0 Flash 38%, and GPT-4o 36% on Hard captions.Claude Sonnet 5 is the only system that improves, from 3.99 to 4.20.
  • Metric behavior: SVG-CLIP-B ranks CLIPDraw second overall on Hard captions, whereas the trained judge keeps CLIPDraw near the floor at 1.28 across all levels.The passage characterizes the CLIPDraw ranking as implausible because CLIPDraw produces abstract strokes for highly complex prompts.

D. Qualitative Results

The qualitative benchmark results compare SVG generators within open-source LLM, painterly rendering, and commercial LLM families across prompts. Each generation is paired with the human-aligned VLM judge’s score, while SVGs that fail to rasterize receive zero.

  • The human-rated dataset visualization orders seven SVGs per prompt by Semantic Alignment score, exposing a broad range from exact matches to missing attributes, relations, or counts.
  • Figures 9–16 show every benchmark generator grouped into open-source LLM, painterly rendering, and commercial LLM families.
  • Each qualitative block renders one prompt with all generators in its family and displays the judge’s score beneath each generation.
  • Non-renderable SVGs are marked explicitly and scored 0 by the judge.

E. Limitations and Scope

The evaluation targets Semantic Alignment rather than the quality of the underlying vector representation, and several benchmark and generalization boundaries remain. Human supervision is limited in density, benchmark rankings rely on automatic evaluators, and reliability may decrease outside the training distribution.

  • What We Measure: The evaluators measure Semantic Alignment of rendered SVGs, not editability, path efficiency, layer organization, or code quality.
  • Annotation Density: Most human-rated examples receive one annotation, while denser re-annotation would provide tighter reliability estimates.
  • Benchmark Supervision: The independent benchmark is not human-annotated; its rankings rely on automatic evaluators validated separately on a held-out human-rated test set.
  • Distribution Shift: Both evaluators may be less reliable for styles or concepts substantially outside their training distribution.
Loading 2609.03806v1…