PaperlayerView exact revision on arXiv

Source-linked AI summary

VectorGym: A Multitask Benchmark for SVG Code Generation, Sketching, and Editing

Juan Rodriguez, Haotian Zhang, Abhay Puri, Tianyang Zhang, Rishav Pramanik, Meng Lin, Xiaoqing Xie, Marco Terral, Darsh Kaushik, Aly Shariff, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli

arXiv:2603.29852v1cs.GRcs.AIcs.CV

TL;DR

SVG evaluation lacks a unified, realistic benchmark covering generation, editing, and understanding with human-authored annotations. VectorGym fills this gap with four tasks and a multi-task reinforcement-learning approach using rendering-based rewards; its trained Qwen3-VL 8B reaches state-of-the-art open-source performance, surpasses larger models, and matches GPT-4o.

  • Problem

    Existing SVG benchmarks lack realistic, unified coverage of visual understanding, vector generation, and structured code manipulation, often relying on synthetic edits and omitting human-authored labels.

  • Method

    VectorGym combines four human-annotated SVG tasks with multi-task reinforcement learning using rendered-image rewards for generation and editing and text-embedding rewards for captioning.

  • Results

    Qwen3-VL 8B achieves state-of-the-art performance among open-source models, surpasses much larger models, and matches GPT-4o; frontier VLM evaluation reveals significant performance gaps.

  • Takeaways & Limitations

    VectorGym establishes a rigorous framework for evaluating and advancing visual code generation across SVG generation, editing, and understanding.

  • Takeaways & Limitations

    The benchmark may inherit biases from evaluated models' training data, affecting fairness and representation across demographics, cultures, and artistic styles.

Abstract

from arXiv · show

We introduce VectorGym, a comprehensive benchmark suite for Scalable Vector Graphics (SVG) that spans generation from text and sketches, complex editing, and visual understanding. VectorGym addresses the lack of realistic, challenging benchmarks aligned with professional design workflows. Our benchmark comprises four tasks with expert human-authored annotations: the novel Sketch2SVG task (VG-Sketch); a new SVG editing dataset (VG-Edit) featuring complex, multi-step edits with higher-order primitives; Text2SVG generation (VG-Text); and SVG captioning (VG-Cap). Unlike prior benchmarks that rely on synthetic edits, VectorGym provides gold-standard human annotations that require semantic understanding and design intent. We also propose a multi-task reinforcement learning approach that jointly optimizes across all four tasks using rendering-based rewards. Our method, built on GRPO with curriculum learning, trains a Qwen3-VL 8B model that achieves state-of-the-art performance among open-source models, surpassing much larger models including Qwen3-VL 235B and matching GPT-4o. We also introduce a VLM-as-a-Judge metric for SVG generation, validated through human correlation studies. Our evaluation of frontier VLMs reveals significant performance gaps, positioning VectorGym as a rigorous framework for advancing visual code generation. VectorGym is publicly available on huggingface.co/datasets/ServiceNow/VectorGym.

1. Introduction

VectorGym addresses the lack of realistic, unified SVG benchmarks by combining four human-annotated tasks with multi-task evaluation and learning methods. It introduces Sketch2SVG and complex human-authored edits while evaluating both open-source and proprietary VLMs.

  • Motivation: Existing SVG evaluations often focus on icons or basic shapes, use synthetic edits, and omit sketch-conditioned generation and human-authored gold labels.This leaves a need for a benchmark spanning visual understanding, vector generation, and structured code manipulation.
  • Benchmark: VectorGym spans Sketch2SVG, SVG Editing, Text2SVG, and SVG Captioning for SVG generation, manipulation, and understanding.The benchmark converts rough sketches to vector code, applies natural-language edits, generates SVGs from text, and describes SVG content.
  • Evaluation: The study evaluates proprietary and open-source frontier VLMs to characterize current capabilities and performance gaps across SVG generation tasks.This analysis provides a broad view of where current models succeed and struggle.
  • Data: The benchmark uses gold-standard human annotations across all tasks and introduces expert-authored edits with complex intent, rich primitives, and non-trivial operations.Its data includes in-the-wild icons, diagrams, emojis, fonts, logotypes, and complex illustrations, with human curation for realistic difficulty.
  • Methods and evaluation: The proposed reinforcement-learning method jointly optimizes all four tasks and achieves state-of-the-art performance among open-source models.The method uses rendering-based rewards and is paired with a task-specific VLM-as-a-Judge evaluation suite validated through human correlation studies.

2. Related Work

Prior work established vector datasets and learning-based SVG generation, but existing methods and benchmarks remain limited in generality and realism. VectorGym responds with human-created, semantically complex edits, a novel sketch task, and human-validated captions.

  • Vector graphics generation: Shape-fitting and learning-based vectorization methods support selected SVG tasks but remain insufficiently general for a wide range of SVG applications.Prior approaches include differentiable rendering, attention architectures, sketch abstraction, and text-conditioned SVG synthesis.
  • VLMs for SVG generation: Modern VLMs frame SVG creation as visual-to-code generation, extending evaluation toward joint visual understanding and program synthesis.StarVector and subsequent work support this direction with structured code generation from visual inputs.
  • SVG datasets and benchmarks: Existing SVG datasets and benchmarks cover icons, understanding, image-to-SVG, text-to-SVG, diagram code generation, and SVG editing from multiple sources.Examples include DeepSVG, FIGR-8, SVG-Stack, UniSVG, VGBench, and SVGEditBench.
  • VectorGym: VectorGym focuses on human-created edits that mirror professional design actions, introduces Sketch2SVG from human sketches, and uses human-validated captions for realistic high-difficulty evaluation.The edits require semantic understanding and are designed to be more complex than synthetic programmatic edits.

VectorGym Benchmark

VectorGym is a multi-task SVG benchmark covering sketch conversion, complex editing, text generation, and captioning with realistic human-authored annotations. It also defines rendering- and language-based evaluation and trains a joint reinforcement-learning model across tasks.

  • Benchmark Tasks: VectorGym evaluates four complementary capabilities: Sketch2SVG, SVG Editing, Text2SVG, and SVG Captioning.Together, these tasks assess visual-to-code generation, structured editing, and SVG understanding.
  • Benchmark Tasks: VG-Sketch converts rough hand-drawn sketches into clean SVG code while testing spatial reasoning, shape recognition, and geometric abstraction.Inputs contain approximate shapes and imperfect lines, requiring structured vector representations.
  • Benchmark Tasks: VG-Edit requires multi-step, semantically guided changes involving complex SVG primitives while preserving validity, visual coherence, and unmodified content.Human-authored examples include adding objects, changing logo content, converting charts, and modifying facial expressions.
  • Dataset Construction: The benchmark uses diverse in-the-wild SVGs spanning icons, diagrams, emojis, fonts, logotypes, and complex illustrations, with curated complexity and human annotation.The collection is sourced from SVG-Stack and retains higher-order SVG structure and primitive detail.
  • Evaluation and Training: VectorGym’s VLM-as-a-Judge protocol evaluates semantic correctness, structural validity, and instruction following, while rendering-based rewards jointly train all four tasks.SVG outputs are rendered and compared with perceptual and pixel-space metrics; captioning uses BGE-M3 embedding similarity.
  • Evaluation and Training: The overall VectorGym score averages task-specific scores across Sketch2SVG, SVG Editing, Text2SVG, and SVG Captioning.Task scores combine VLM-Judge with task-specific visual, embedding, or text metrics before arithmetic aggregation.

4. Experiments

The experiments evaluate frontier proprietary and open-source VLMs across all four VectorGym tasks using in-context learning and joint reinforcement-learning training. The setup emphasizes broad model coverage and fair comparison while exposing SVG-specific challenges.

  • Experimental Setup: The evaluation covers all four VectorGym tasks and is designed to compare models fairly while highlighting SVG code-generation challenges.The benchmark evaluates sketch, editing, text-generation, and captioning capabilities.
  • Model Coverage: The study evaluates both proprietary and open-source frontier VLMs with code-generation capabilities.Model selection follows a systematic approach intended to cover the current landscape.
  • Experimental Setup: In-context learning experiments use strong task prompts to assess frontier-model capabilities.The comparison includes open- and closed-source models under prompts specified in the appendix.
  • Reinforcement Learning: The reinforcement-learning experiment trains Qwen3-VL 8B Instruct jointly across all four tasks using rendered SVG outputs for reward computation.This training uses the VectorGym training split and the RLRF framework with GRPO.

5. Results

VectorGym evaluation reveals substantial capability differences across SVG modalities, with Gemini 3 Pro leading overall while the trained Qwen3VL 8B Gym model competes strongly with larger and proprietary models.

  • Sketch2SVG: 78.56: Gemini 3 Pro achieves the highest Sketch2SVG Score, while GPT-5.1 follows with 75.69.Gemini 3 Pro also records a VLM Judge score of 72.20.
  • Sketch2SVG: 70.72: Qwen3VL 8B Gym surpasses GPT-4o at 69.55 and Qwen3VL 235B at 67.52 on Sketch2SVG Score.Its VLM Judge score rises to 46.00 from 33.00 for base Qwen3VL 8B Instruct.
  • SVG Editing: 82.81: Qwen3VL 8B Gym exceeds GPT-4o at 82.35 on SVG Editing Score, while Gemini 3 Pro leads at 88.71.The Gym model’s MSE is 8.36 versus 11.01 for the base 8B model.
  • Text2SVG: 70.68: GPT-5.1 leads Text2SVG among proprietary models by VLM Judge score, while Qwen3VL 8B Gym scores 63.60 versus 59.68 for Qwen3VL 235B.Qwen3VL 8B Gym reaches parity with GPT-4o at 63.09 on the reported open-source comparison.
  • SVG Captioning: Qwen3VL 8B Gym leads BGE-M3 and ROUGE captioning metrics, but Gemini 3 Pro leads VLM Judge, revealing metric-dependent rankings.The Gym model records BGE-M3 79.76, ROUGE 25.58, and VLM Judge 35.80.
  • Overall Performance Hierarchy: Gemini 3 Pro leads overall with a VectorGym score of 73.17, followed by GPT-5.1 at 71.36.

6. Conclusion

VectorGym provides a four-task, human-annotated benchmark and a multi-task reinforcement-learning method for SVG code generation and manipulation. The evaluation finds significant performance gaps, with Gemini 3 Pro leading overall, while the benchmark supports more rigorous visual-code-generation evaluation.

  • VectorGym covers Sketch2SVG, SVG editing, Text2SVG, and SVG captioning with gold-standard human annotations.
  • The proposed method uses rendering-based rewards to train a Qwen3-VL 8B model, achieving state-of-the-art performance among open-source models.
  • Frontier VLM evaluation reveals significant performance gaps, with Gemini 3 Pro leading overall.
  • The benchmark establishes a rigorous evaluation standard and framework for advancing SVG generation capabilities.
  • Models may exhibit training-data biases affecting fairness and representation across demographics, cultures, and artistic styles.

A. VectorGym Data Creation

VectorGym’s data creation combines curated SVG sources, expert review, filtering, split controls, and task-specialized annotation vendors. The process is designed to produce realistic, diverse, and computationally manageable examples.

  • 7,000 high-quality SVG samples were extracted through a rigorous multi-stage filtering process.
  • Human experts filtered samples for visual appeal and well-formedness, removing corrupted, overly simplistic, or poorly designed graphics.
  • A 2,000–8,000-token length range preserved meaningful complexity while maintaining computational feasibility.
  • Color entropy thresholding removed samples with insufficient visual diversity or monotonic palettes.
  • Random sampling was used to reduce systematic biases in content distribution.
  • The dataset retained 300 SVG-Stack test samples for testing and selected 100 training samples for validation and metric-design studies.
  • Two specialized vendors handled sketch and caption generation or technical SVG editing, supported by more than 20 trained annotators.

A.1.4. TASK-SPECIFIC ANNOTATION PROCEDURES

Task-specific annotation procedures create varied sketch, caption, and complex-edit examples using professional tools, validation, and consistency checks. The procedures emphasize realistic visual inputs and edits requiring semantic understanding.

  • Sketch2SVG Generation: Sketch2SVG examples include chart-type conversions and structural modifications requiring new geometric primitives.
  • Sketch2SVG Generation: Sketch annotations include hand-drawn and digital variants, with both colored and black-and-white versions for modality robustness.
  • SVG Editing: SVG editing excluded simple operations such as rotation, recoloring, scaling, and basic removal to avoid trivial edits.
  • SVG Editing: Required edits include path modifications, primitive additions, parameter adjustments, and conceptual additions requiring semantic understanding.
  • Caption Generation: Caption generation begins with detailed visual descriptions emphasizing accurate color specification.
  • Caption Generation: Qwen2-VL-32B cross-validated human descriptions for consistency and completeness before instruction reformatting.
  • Caption Generation: Caption instructions were produced in hexadecimal-color and natural-language-color variants, with hexadecimal specifications empirically yielding higher SVG-generation accuracy.
  • Quality Assurance: All annotations underwent automated syntax validation, human verification, and consistency checks across related task pairs.

A.2. VLM-as-Judge Evaluation Metric for SVG Generation

The paper develops task-specific VLM-as-a-Judge metrics because existing SVG evaluation methods miss nuanced visual and semantic qualities. It validates candidate judges against expert human ratings and selects judges separately for different tasks.

  • Existing SVG metrics often fail to jointly capture visual fidelity and semantic alignment in vector-code outputs.
  • The evaluation framework uses task-specific prompts and 0–5 VLM scores focused on visual accuracy, semantic alignment, code quality, and aesthetics.
  • The study compares multiple open- and closed-source VLM judges on generations from Claude 4.5 Sonnet, Gemini 3 Pro, and GPT-4o using 100 validation samples.
  • Human validation involved 17 technical or AI and design experts who produced around 674 ratings for correlation analysis.
  • Gemini 3 Pro is selected for VG-Sketch, VG-Cap, and VG-Text, while GPT 5.1 is selected for VG-Edit based on correlations with human judgments.

B. Additional Qualitative Results

The qualitative analyses illustrate how models handle sketch-to-SVG, text-to-SVG, and SVG-editing tasks, emphasizing semantic fidelity, geometric representation, and instruction adherence. They also specify evaluation criteria that prioritize visible content and penalize unsupported or contradictory details.

  • The qualitative results cover Sketch2SVG, Text2SVG, and SVG editing, with examples comparing inputs, ground truth, instructions, and generated outputs.
  • Text2SVG: Text2SVG examples show both accurate semantic and geometric generations and failures involving primitive choice, semantic misunderstanding, and incomplete shapes.
  • Benchmark coverage: The overview identifies four benchmark tasks: Sketch2SVG, SVG Editing, Text2SVG, and SVG Captioning.
  • Sketch2SVG: Sketch2SVG evaluation emphasizes object presence, shapes, sizes, colors, spatial relations, and preservation of the input sketch’s visual content.
  • Sketch2SVG: The Sketch2SVG rubric penalizes hallucinated additions more than small omissions and penalizes altered key elements more than minor rendering differences.
  • SVG Editing: Editing evaluation prioritizes semantic similarity and visual preservation while treating image quality, clarity, and coherence as secondary factors.

D. Captioning Metrics

Captioning quality is evaluated by averaging several pairwise reference–prediction metrics across the corpus. The metric suite combines lexical overlap, semantic similarity, and rubric-based language-model judgment.

  • Captioning metrics are computed pairwise over aligned reference and prediction captions, then averaged across the corpus.
  • BLEU measures n-gram precision with a brevity penalty, while CHRF++ measures character n-gram F-score and ROUGE-L measures longest-common-subsequence overlap.
  • BERTScore measures semantic similarity through contextual embeddings, and BGE-M3 Similarity uses average cosine similarity between sentence embeddings.
  • GPT-5 Rubric Similarity uses LLM-judged semantic agreement on a 0–5 rubric mapped to 0–100.

E. Data Licensing

VectorGym’s SVG data derives from SVG Stack and The Stack, so its licensing and governance follow The Stack’s source-data framework. The stated source policy restricts included repositories to permissive licenses and propagates opt-out removals.

  • All SVG data originate from SVG Stack and The Stack, causing VectorGym to inherit The Stack’s copyright and license characteristics.
  • The Stack includes repositories under permissive licenses such as MIT, Apache, BSD, and CC0 while excluding non-permissive or non-redistributable licenses.
  • Developer opt-out removals are automatically propagated to derived datasets, and SVG Stack retains original file paths and license identifiers.
  • The authors use SVG Stack as distributed without external sources and intend to release the processed subset under compatible original licensing terms.
Loading 2603.29852v1…