PaperlayerView exact revision on arXiv

Source-linked AI summary

VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran, Dantong Zhu, Dongxing Mao, Linjie Li, Philip Torr, Alex Jinpeng Wang

arXiv:2511.02778v1cs.CVcs.CL

TL;DR

Visual-centric coding remains underexplored despite SVG’s potential as a compact symbolic representation of natural images. VCode evaluates image-to-SVG generation through reasoning over rendered code, and VCoder gains +12.3 overall over top-performing Claude-4-Opus while professional knowledge and 3D reasoning remain difficult.

  • Problem

    Visual-centric coding is underexplored relative to language-centric coding, despite SVG’s compact, interpretable, executable form for representing natural-image structure.

    Source: §1 · p. 2

  • Method

    VCode recasts multimodal understanding as SVG generation and tests symbolic fidelity with questions over rendered SVGs; VCoder adds iterative revision and visual tools.

    Source: §1 · p. 2

  • Results

    +12.3 overall gain over top-performing Claude-4-Opus; frontier VLMs nevertheless remain limited in professional disciplines and 3D perception.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Human studies underscore the potential of symbolic visual coding as a pathway toward more human-aligned multimodal intelligence.

    Source: §6 · p. 9

    Coders struggle to capture fine-grained image details such as boundaries, motivating visual tools that provide complementary metadata.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks such as program synthesis and debugging, leaving visual-centric coding underexplored. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream reasoning. VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVGs; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues such as objects, shapes, and text beyond the model's intrinsic capacity. Across benchmarks, frontier VLMs with strong reasoning capabilities score well overall yet remain limited in professional knowledge and 3D reasoning. VCoder delivers a 12.3-point overall gain over the top-performing Claude-4-Opus. Human studies show that both humans and VLMs perform worse on rendered SVGs, their consistency reveals the promise of symbolic visual representation. The benchmark and code are available at https://github.com/CSU-JPG/VCode.

1 Introduction

VCode recasts multimodal understanding as generating SVG that preserves natural-image meaning, and CodeVQA tests that fidelity through questions answered over rendered SVGs. VCoder adds iterative revision and visual tools, while experiments and human studies reveal both persistent challenges and the promise of symbolic visual coding.

1 Introduction

  • Prior coding benchmarks emphasize linguistic tasks, while code-generated visual assets such as charts and diagrams are not grounded in the natural visual world.The paper positions VCode as addressing natural images rather than only synthetic visual assets.Source: §1 · pp. 1–2
  • SVG offers a compact, interpretable, executable alternative to dense pixels, reflecting how sketches foreground counts, spatial relations, and structural outlines.This motivates asking whether visual representation can move beyond pixels toward code for representing and reasoning about images.Source: §1 · p. 2
  • VCode reframes multimodal understanding across general, college-level, and visual-perception domains as generating SVG, with CodeVQA testing questions answered from rendered code.The benchmark repurposes MM-Vet, MMMU, and CV-Bench tasks; Table 1 characterizes its task, scope, and Render → VQA evaluation.Source: §1 · p. 2, Figure 1 · p. 2, Table 1 · p. 3
  • +12.3 overall gain over top-performing Claude-4-Opus comes from VCoder’s iterative SVG revision and structured cues from visual tools.Revision compares intermediate renderings with the original, articulates discrepancies, and updates the SVG; tools supply cues such as objects, shapes, and text.Source: §1 · p. 2
  • Human studies find patterns consistent between humans and VLMs reasoning over rendered SVGs and raw images, suggesting potential for symbolic visual coding.The paper presents this consistency as suggesting promise, not as evidence that SVGs outperform raw images.Source: §1 · p. 3

2 Related Works

Prior coding benchmarks focus on language-centric code tasks or synthetic visual assets, while multimodal understanding benchmarks generally interact through natural language. VCode addresses the open question of encoding natural images as executable visual code.

2.1 Coding Benchmarks

  • Language-centric coding benchmarks evaluate program correctness and execution, from synthesis and debugging to software engineering in large repositories.The section cites HumanEval [4], MBPP [5], and SWE-Bench [8] as examples.Source: §2.1 · p. 3
  • Visual coding benchmarks cover charts, user interfaces, and vector graphics, but largely leave natural images unaddressed.VCode repurposes multimodal understanding tasks to study SVG coding grounded in natural images.Source: §2.1 · p. 3

2.2 Multimodal Understanding

  • Multimodal benchmarks target general or professional reasoning, but typically use natural-language queries or answers rather than symbolic visual code.VCode frames image-to-code generation as distilling concepts and structure into a symbolic format bridging perception with reasoning and action.Source: §2.2 · pp. 3–4

3 VCode Benchmark

VCode casts image understanding as SVG generation that preserves symbolic information, then evaluates whether the rendered code supports reasoning across commonsense, professional, and visual-perception tasks.

3.1 Task Definitions

  • Given an image, a model generates SVG whose rendered output should minimize the discrepancy in symbolic information from the original image.The paper frames the objective around symbolic rather than pixel-level correspondence.Source: §3.1 · p. 4

3.2 Evaluation Metrics

  • Evaluation combines embedding consistency, question answering over the rendered SVG, and generated-code token count.CodeVQA tests whether the rendered image alone supports correct answers; the SigLIP score compares visual embeddings, while token count measures code concision.Source: §3.2 · p. 4

3.3 Data Curation

  • VCode repurposes MM-Vet, MMMU, and CV-Bench, yielding 218, 146, and 100 image-question pairs respectively, or 464 pairs total.The CV-Bench subset is stratified across spatial relationship (20), object count (20), depth order (30), and relative distance (30).Source: §3.3 · pp. 4–5

4 VCoder

VCoder addresses challenges in long-context SVG generation, unseen render feedback, and fine visual detail with iterative revision and external visual cues.

4 VCoder

  • VCoder pairs test-time SVG revision with visual tools that provide structured cues for generation.The approach targets long code sequences, cross-modal outputs whose visual effects are unseen until execution, and difficulty encoding fine details such as edges, masks, and colors.Source: §4 · p. 5

4.1 Thinking with Revision

  • The revision loop detects discrepancies between the rendered output and target image, then updates the SVG using the difference signal and current code and rendering.The loop repeats across iterations to progressively refine the reconstruction.Source: §4.1 · pp. 5–6

4.2 Act with Visual Tools

  • Visual tools address fine-grained attribution limits by supplying object categories, bounding-box locations, segmentation-derived contours, and detected text.SAM-2 masks are simplified into sparse polygon coordinates while keeping area nearly unchanged; OpenOCR text is encoded with SVG's native text tag.Source: §4.2 · p. 6

5 Experiments

Experiments show that VCoder improves SVG-based multimodal performance, while results expose persistent challenges in professional knowledge, 3D perception, and generating faithful visual code. Ablations and qualitative examples examine visual tools, revision, input modality, and human reasoning over rendered SVGs.

5.2 Main Results

  • GPT-5 leads the baselines with SigLIP 72.3 and CodeVQA overall 46.8, but the best SVG result remains below the original-image upper bound of 61.7.Knowledge is consistently the lowest MM-Vet dimension; MMMU scores cluster in a modest band, and CV-Bench is difficult especially for 3D relations.Source: §5.2 · pp. 7–8
  • 12.3 points: VCoder raises Claude-4-Opus’s Overall score from 41.7 to 54.0, improving all three domains through revision and visual-tool assistance.The gain is reported for the full VCoder framework.Source: §5.2 · p. 8
  • Longer SVG outputs correlate with higher scores, but length alone is insufficient; usable context remains a visual-coding bottleneck.Short outputs such as Qwen-2.5-VL’s 0.3K tokens underperform, while GPT-5 and Gemini-2.5-Pro often produce sequences exceeding 2K tokens.Source: §5.2 · p. 8

5.3 Key Ablations

  • All vision tools improve performance by 16.6 points over Claude-4-Opus; shape cues particularly help spatial reasoning, and the full ensemble yields the largest overall improvement.The ablation examines location, category, shape, and text cues.Source: §5.3 · p. 8, Table 3 · p. 8
  • Figure 4 compares policies on original images and SVGs: scores fall on SVGs, while humans and VLMs show alignment when interpreting symbolic representations.Human performance decreases from 50.4 on original images to 40.6 on SVGs; GLM-4.5V scores 75.5 on original images.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Revision benefits Claude and GLM-4.5V in the first round, while GPT-4o improves in later rounds after initially struggling.The authors suggest that effective revision depends on a strong reasoning foundation.Source: §5.3 · p. 8, Figure 5 · p. 9
  • Captioning an image before SVG generation performs best among the tested input modes, while direct image-to-SVG remains weak even with deep thinking.Table 4 compares different input modes for Claude-4-Opus.Source: §5.3 · p. 8, Table 4 · p. 9

5.4 Qualitative Analysis

  • Figure 6 illustrates how visual tools recover key geometry and revision refines details, while MMMU and CV-Bench examples expose knowledge-intensive and spatial challenges.The examples include a multi-panel historical relief and scenes requiring object localization and spatial relations.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Conclusion

The conclusion presents VCode and CodeVQA as a benchmark and evaluation protocol for symbolic fidelity, and VCoder as an approach combining revision with visual tools. It identifies end-to-end vision–language coders trained with scalable data as future work.

6 Conclusion

  • VCode benchmarks multimodal understanding through SVG representations, while CodeVQA tests symbolic fidelity by asking questions over rendered SVGs.The conclusion reports that frontier VLMs struggle to produce faithful SVGs despite strong linguistic reasoning, and that VCoder improves performance through revision and visual tools.Source: §6 · p. 9
  • The authors propose end-to-end vision–language coders with scalable training data as a direction for producing more faithful symbolic representations.The conclusion describes human studies as underscoring the potential of symbolic visual coding for more human-aligned multimodal intelligence.Source: §6 · p. 9

A Implement Details

  • The implementation uses an NVIDIA RTX 4090 with a maximum output length of 16,384 tokens, reduced to 8,192 for Qwen2.5-VL models.The model is implemented using PyTorch.Source: A Implement Details · p. 13
  • Evaluation uses gpt-4-0613 for MM-Vet and rule-based string matching for CV-Bench and MMMU; SigLip2 uses the so400m-patch14-384 model.Reported token costs are measured with the tiktoken cl100k base encoding; StarVector does not accept textual prompts in img2svg experiments.Source: A Implement Details · p. 13
  • Test-time Revision iterates over an image, its initial rendering, and initial SVG code to return a refined rendering.Each iteration generates revised SVG code and renders it again.Source: A Implement Details · p. 13

B.1 Effects by SigLip v.s. DINO

The appendix contrasts visual feature extractors and identifies revision-round results on MM-Vet as a separate evaluation. Its supplied text describes the comparison scopes but does not report their measured outcomes.

B.1 Effects by SigLip v.s. DINO

  • DINO emphasizes low-level visual representation, whereas SigLip2 emphasizes semantic space, according to the comparison accompanying Table 5.The table caption says best and second-best results are highlighted and underlined, respectively.Source: Table 5 · p. 14

B.2 Effect by revision round on MM-Vet

  • Table 6 reports MM-Vet results by revision round for each model, with best and second-best results highlighted and underlined.The supplied passage does not state the scores or identify which rounds perform best.Source: Table 6 · p. 14

B.3 Effects by different policy during evaluation on MM-Vet

This appendix includes a policy-evaluation table for MM-Vet, a two-stage caption-to-SVG prompting procedure, and prompts specifying visual-tool metadata integration and SVG output requirements.

B.3 Effects by different policy during evaluation on MM-Vet

  • Table 7 compares evaluators on original-image and VCoder settings for MM-Vet, marking each setting’s best and second-best results.The supplied caption does not give the scores or identify the winning evaluator.Source: Table 7 · p. 14

C.2 Img2Text2SVG

  • Img2Text2SVG first prompts for a detailed image caption covering objects, appearance, spatial relations, style, and text, then uses that description to generate SVG code.The generation prompt requests complete, self-contained SVG with appropriate dimensions and precise shapes, colors, and positioning.Source: §C.2 · p. 15

Visual Tools – System Prompt

  • The visual-tools system prompt treats OCR quadrilaterals and detected object paths as authoritative placement and contour cues, and allows identity hints for recognizable subjects.It also directs reconstruction of unlabeled regions using native SVG primitives.Source: Visual Tools – System Prompt · pp. 16–17

Visual Tools – User Prompt

Revision – Stage 1: Visual Difference Analysis

The revision stage analyzes visual differences between the original image and rendered SVG to guide code updates. Qualitative comparisons report that VCoder more faithfully reconstructs visual content while preserving semantic consistency.

Revision – Stage 1: Visual Difference Analysis

  • Revision analysis identifies localized differences, visual-attribute mismatches, and specific SVG edits to improve the rendering.Suggested edits include changing SVG elements and attributes such as fill, stroke, and coordinates.Source: Revision – Stage 1: Visual Difference… · p. 17

D.1 VCoder vs. Baselines

  • VCoder’s examples show more faithful visual reconstruction and semantic consistency than the two strongest baselines.The comparisons cover representative examples from three benchmarks.Source: §D.1 · p. 18

D.1.1 MM-VET

The qualitative examples span general-commonsense, college-discipline, and visual-perception questions. They include image-based factual questions as well as questions about object counts, depth, and spatial relationships.

D.1.1 MM-VET

  • MM-Vet examples include a question about an instruction’s third step and a short introduction to The Starry Night.The cited examples give “milk” as the third-step answer and describe the painting’s subject and provenance.Source: §D.1.1 · pp. 18–19

D.1.2 MMMU

  • MMMU examples include questions about lymphocyte-transfer experiments, signal transduction, and art-historical topics.The passages show question prompts but do not provide a supported overall outcome for these cases.Source: §D.1.2 · pp. 20–22

D.1.3 CV-Bench

  • CV-Bench examples ask about person counts, relative depth, and the position of one object relative to another.The questions include comparing a table with a bookcase and locating a bottle relative to a wine glass.Source: §D.1.3 · pp. 22–24

D.2 VCoder Individual Components

The ablation visualizations present outputs progressively, from an unrefined rendering through visual-tool use and revision. Other examples show responses to procedural, factual, and equation questions.

D.2 VCoder Individual Components

  • The ablation examples compare the original image with an initial rendering, a tool-assisted output, and a final revised output.The passage says the progressive visualizations demonstrate incremental improvements in image quality and accuracy.Source: §D.2 · p. 24
  • Procedural examples include answers about adding milk, hopping after snapping fingers, and the step for adding flour.The recorded answers are “milk,” “hop on one foot,” and “third / 3.”Source: §D.2 · pp. 25–26
  • Other examples include a painting description and answers to questions about an equation and an object’s location.The painting example identifies Girl With A Pearl Earring; another answer says conditioner is to the right of shampoo.Source: §D.2 · pp. 25, 27
Loading 2511.02778v1…