PaperlayerView exact revision on arXiv

Source-linked AI summary

SketchVLM: Vision language models can annotate images to explain thoughts and guide users

Brandon Collins, Logan Bolton, Hung Huy Nguyen, Mohammad Reza Taesiri, Trung Bui, Anh Totti Nguyen

arXiv:2604.22875v2cs.CVcs.AI

TL;DR

Text-only VLM answers can be difficult to verify, while people often annotate images to explain their reasoning. SketchVLM adds editable, non-destructive SVG overlays to input images; it outperforms sketching models by +28.5 percentage points in accuracy and +48.3% in annotation quality.

  • Problem

    Modern VLMs typically answer image questions with text that can be difficult to verify, unlike people who annotate images to explain their reasoning.

    Source: §1 · pp. 1–2

  • Method

    SketchVLM enables VLMs to produce editable, non-destructive SVG annotations grounded on the input image, without task-specific training.

    Source: §1 · p. 2

  • Results

    +28.5 percentage points in accuracy and +48.3% in annotation quality: SketchVLM outperforms sketching models on both measures.

    Source: §6 · p. 18

  • Takeaways & Limitations

    SketchVLM's visual explanations let users verify model reasoning at a glance.

    Source: §6 · p. 18

    SketchVLM does not perform well on small VLMs that struggle with instruction following.

    Source: §6 · p. 18

Abstract

from arXiv · show

When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.

1 Introduction

SketchVLM lets vision-language models explain image-based answers with editable visual annotations, addressing the difficulty of verifying text-only responses. The introduction reports stronger annotation generalizability, accuracy, and quality than specialized sketching models, alongside findings on image preservation and interaction settings.

1 Introduction

  • Text-only VLM answers can be difficult to verify, whereas people often point, circle, underline, label, or annotate images to explain their reasoning [55].The introduction gives checking a car’s oil level as an example of a task where a visual annotation is easier to verify than a paragraph.Source: §1 · pp. 1–2
  • SketchVLM generates SVG annotations in a separate overlay on the original image, without modifying it or requiring task-specific training.The paper tests the framework with multiple VLM backbones.Source: §1 · p. 2
  • Across seven tasks, SketchVLM is evaluated on drawing and visual reasoning, with examples showing more accurate trajectories, dot connections, and maze paths.The drawing tasks are connecting-the-dots, part labeling, and drawing shapes around objects; the reasoning tasks cover physics, counting, and maze navigation.Source: §1 · pp. 2–3
  • SketchVLMs based on frontier models produce annotations with superior generalizability, accuracy, and quality compared with specialized fine-tuned sketching models.The paper reports these comparisons across its evaluation results.Source: §1 · p. 3
  • Nano Banana Pro cannot generate a separate overlay and frequently alters the source image, while fine-tuned specialist VLMs often fail to generalize to new tasks.The introduction also illustrates undesirable image changes and implausible ball-drop and maze trajectories from Nano Banana.Source: §1 · p. 3
  • SketchVLMs have similar accuracy in single- and multi-turn settings, single-turn generation is faster, and an external x–y coordinate grid improves drawing and question-answering accuracy but is unnecessary.The grid improves drawing capability and question-answering accuracy; the introduction does not state that it is required.Source: §1 · p. 4

2 Related Work … 6. VPCT

SketchVLM is a training-free harness that lets VLMs generate editable SVG annotations directly over input images. The supplied evaluation passages describe benchmark tasks and datasets, but do not report outcome comparisons or scores.

2 Related Work

  • SketchVLM differs from prior tool-based, visual-sketching, and fine-tuned approaches by using state-of-the-art VLMs to create editable SVG overlays directly on input images.The comparison table defines vector overlays as structured, non-destructive marks aligned to an image or canvas, distinct from pixel-space image edits.Source: §2 · p. 5, Table 1 · p. 5

3 SketchVLM

  • SketchVLM combines coordinate-grid visual prompting, structured stroke-output instructions, and XML-to-SVG conversion to place annotations over the source image.It supports rectangles, arrows, text labels, straight lines, and Bézier curves; the model annotates its reasoning before giving a final answer.Source: §3 · pp. 5–6

1. Connect-the-Dots

  • Connect-the-dots evaluation uses 100 images across random dot patterns, silhouette-derived puzzles, and worksheet-style examples, requiring dots to be connected in order.The three subsets contain 21, 30, and 49 images, respectively.Source: §1 · p. 6

2. Counting Objects

  • Object-counting evaluation uses 746 CountBench and Pixmo-Count images, requiring models to count target objects and place numbered markers on them.Counts range from 0 to 10; unsuitable Pixmo-Count examples are filtered out.Source: §2 · p. 6

3. Drawing Shapes around Objects

  • Drawing-shapes evaluation uses 1,000 COCO validation images, with models localizing objects by drawing rectangles or ovals.Images are selected to balance object count and object size across classes.Source: §3 · p. 6

4. Part Labeling

  • Part-labeling evaluation uses 985 PACO and Pascal-Part images across 52 classes, requiring text labels at the corresponding part locations.Images have one target object occupying at least 10% of image area and at least four annotated part labels.Source: §4 · p. 6

5. Maze Navigation

  • Maze evaluation uses 200 generated 3 × 3 mazes with shortest paths of 3–8 steps, requiring models to trace a path and judge whether it reaches the goal without crossing walls.Invalid paths are generated by perturbing one direction in a ground-truth path.Source: §5 · p. 6

6. VPCT

  • VPCT evaluation uses 100 hand-crafted images and asks models to predict which container a dropped ball will land in.The passage identifies the Visual Physics Comprehension Test as the benchmark.Source: §6 · p. 6

7. Ball Drop

The ball-drop task tests whether models can predict a landing container and trace a ball’s trajectory. The paper reports that SketchVLMs produce visual reasoning traces while maintaining competitive accuracy.

7. Ball Drop

  • The ball-drop benchmark contains 198 synthetic images with equal numbers of 1, 2, and 3 randomly placed lines; models choose among four containers and trace the ball’s trajectory.The ball’s horizontal position is randomized, and ground-truth trajectories are produced using PHYRE [4].Source: §7 · p. 7

4.2 Setup

  • Evaluation compares single-turn annotation and answering with multi-turn generation of one stroke per turn, and tests coordinate-grid prompting against normalized 1000×1000 coordinates.In multi-turn generation, the model sees prior annotations rendered on the image and their text representations, then gives its final text answer on the last turn.Source: §4.2 · p. 7

4.3 Baselines

  • The baselines include image-editing approaches, Qwen-2.5-VL-7B fine-tuned for SVG annotations [3], BAGEL-7B-MoT fine-tuned for image editing [15], and text-only VLMs.Source: §4.3 · pp. 7–8

4.4 Metrics

  • Performance is assessed by answer accuracy, annotation–text alignment, and annotation quality, with quality judged on a 1-to-5 rubric for plausibility and visual clarity.Alignment measures how often a VLM judge infers the same answer from the annotation as the model gives in text.Source: §4.4 · p. 8
  • SketchVLMs produce visual reasoning traces while maintaining competitive accuracy, according to Table 2.Source: Table 2 · p. 9

5 Results

Across reasoning and annotation tasks, SketchVLM produces useful visual explanations, with strong task performance and annotations that align with answers. Results also show that input grids, drawing formats, and interaction settings affect performance.

5.2 SketchVLMs can localize points and connect them in order

  • SketchVLM can localize up to 35 points with RMSE 5.92, and its reported point-ordering accuracy is 74% and 99%, respectively.Adding a grid degrades Connect-the-Dots localization, increasing RMSE from 5.92 to 99.34, although the grid boosts spatial precision in other evaluated tasks.Source: §5.2 · pp. 9–10, sketchVLM · p. 9

5.3 SketchVLM improves counting accuracy

  • Reported counting results include 94.5 counting accuracy with 95.9 numeric-marker location accuracy, while another result pairs 75.4 counting accuracy with 51.0 marker accuracy.Numeric markers let users visually verify counts; the lower marker accuracy shows that correct counts do not always correspond to correctly placed markers.Source: §5.3 · pp. 10–11

5.4 SketchVLMs localize objects more accurately with pre-defined shape primitives than with free-form annotations

  • Stroke-based rectangles slightly improve medium-object AP50 (+0.6) and large-object AP50 (+1.4), but reduce small-object AP50 (-10.1) and overall AP50 from 63.1 to 58.8.SketchVLM matches the original model in precision but has lower recall; the paper proposes combining predefined shapes with free-form strokes.Source: §5.4 · p. 12

5.5 SketchVLM improves localization accuracy for when labeling parts of an object

  • For part labeling, SketchVLM’s accuracy gap narrows from -3.8 at r = 0 to -0.6 at r = 7, as boundary tolerance increases.Remaining errors are dominated by wrong-position mistakes (>79%); the reported differences largely reflect small boundary offsets rather than semantic labeling failures.Source: §5.5 · p. 13

5.6 SketchVLMs outperform models fine-tuned directly on path-tracing tasks

  • SketchVLM variants achieve maze-navigation accuracies of 98.0% and 92.8%, while models fine-tuned for path tracing score near the 50% random-choice baseline.A model with 93.3% accuracy sometimes alters the entire image; SketchVLM’s annotations aid user verification even when its accuracy gain over the default model is minor.Source: §5.6 · p. 13

5.7 Fine-tuned sketching models fail to generalize to unseen physics understanding tasks

  • SketchVLM reaches 96.0% accuracy on VPCT and 79.7% on Ball Drop, while fine-tuned sketching models remain near random on both tasks.The fine-tuned models score 37% and 27% on VPCT (random baseline 33.3%) and 35.9% and 30.3% on Ball Drop (random baseline 25.0%).Source: §5.7 · p. 14

5.8 SketchVLM has higher annotation–text alignment than image-editing and fine-tuned models

  • SketchVLM has the highest annotation–text alignment and annotation quality, with mean quality scores of 3.70 from a VLM judge and 4.14 from human raters.The judge’s ratings correlate positively with human judgments but show only moderate agreement, so they serve as a proxy to consider alongside task accuracy and alignment.Source: §5.8 · p. 15, sketchVLM has significantly higher… · p. 16, §5.10 · p. 17

5.11 Single-turn is as accurate as multi-turn but requires significantly fewer turns

  • Single-turn generation achieves comparable or higher accuracy than multi-turn across tasks while requiring about 5.92x fewer turns.Removing the text representation of prior strokes notably degrades annotation quality, so reported multi-turn results include both rendered images and text history.Source: §5.11 · p. 17

6 Conclusion

SketchVLM uses a training-free framework to produce editable, non-destructive SVG annotations grounded in input images, helping users verify VLM reasoning. It outperforms sketching models in accuracy and annotation quality, but performs poorly on small VLMs that struggle with instruction following.

6 Conclusion

  • +28.5 percentage points in accuracy and +48.3% in annotation quality over sketching models, while generalizing to real-world tasks.The annotations are visual explanations intended to let users verify model reasoning at a glance; text-only responses and image-editing baselines do not reliably provide this.Source: §6 · p. 18

Limitations and Future Work

  • SketchVLM can transfer to strong open-source VLMs like Kimi K2.5 [44], but performs poorly on small VLMs that struggle with instruction following, such as Qwen2.5-VL-7B [3].Source: §6 · p. 18
  • Enabling models to undo and edit strokes is a potential way to improve multi-turn performance.Source: §6 · p. 18

7 Author contribution statement

Brandon Collins, Logan Bolton, and Hung Nguyen led benchmark creation or curation and experiments, while the team shared writing, review, demo, website, and supervision responsibilities.

7 Author contribution statement

  • BC, LB, and HN were identified as major contributors to benchmark dataset creation or curation and experiment execution.Source: §7 · p. 18
  • HN curated three benchmarks and led their evaluation and the human-versus-VLM agreement study; BC created Connect-the-Dots and led method and evaluation work.BC also led inference and evaluation code and ablations, including evaluation on Connect-the-Dots and VPCT.Source: §7 · p. 18
  • LB created the Ball Drop and Maze Navigation benchmarks, led their evaluation, ran open-source model experiments, and led annotation-quality and annotation-text alignment evaluations.Source: §7 · p. 18
  • BC and LB led manuscript writing and developed the demo with AN; all authors edited and reviewed, BC led the website, LB contributed, and AN supervised the project.BC and LB were technical team leads.Source: §7 · p. 18

A Real World Applications … C.1 Combined Results

The appendix illustrates real-world visual guidance and documents datasets for six tasks; combined results report gains in visual-reasoning accuracy and annotation quality over alternative sketching approaches.

A Real World Applications

  • Real-world examples show visual guidance for setting up an AWS EC2 instance, plus multi-turn help with Docker, GitHub CI/CD, and other use cases.One figure contrasts SketchVLM’s visual annotations with ChatGPT’s text-only response.Source: Figure A1 · p. 25, Figure A2 · p. 25, Figure A3 · p. 26

B.1 Connect-the-Dots

  • The connect-the-dots benchmark contains 100 images across random-dot, outline, and worksheet subsets.The subsets include 21 generated random-dot images, 30 converted outlines, and 49 collected worksheets; worksheet dot coordinates were manually annotated.Source: §B.1 · p. 27

B.2 Counting

  • The counting dataset combines CountBench and TallyQA samples with Pixmo-Count, covering object counts from 0 to 10.CountBench and TallyQA together contribute 746 samples; Pixmo-Count contributes 443 after unsuitable cases are removed.Source: §B.2 · p. 27

B.3 Drawing Shapes around Objects

  • The drawing-shapes test set uses 1,000 COCO validation images selected to balance object counts across classes and sizes.The size categories are small, medium, and large.Source: §B.3 · p. 27

B.4 Part Labeling

  • The part-labeling dataset contains 985 images across 52 classes, selected for a single target object, sufficient size, at least four part labels, and class balance.Images were selected from PACO and Pascal-Part; each target object occupies at least 10% of the image area.Source: §B.4 · p. 27

B.5 Maze Navigation

  • The maze benchmark uses 200 unique 3×3 grids, with shortest paths of 3–8 steps and a direction command randomly changed to create an invalid path.The task is to determine whether a commanded path from the green start to the red goal reaches the goal without crossing border walls.Source: §B.5 · p. 28

B.6 Ball Drop

  • The Ball Drop benchmark provides simulated ground-truth trajectories for 198 images with one to three lines and four containers; VPCT lacks trajectory ground truth.VPCT has 100 hand-crafted images and asks which of three buckets receives a dropped ball; Ball Drop uses PHYRE simulations and four containers.Source: §B.6 · p. 28

C.1 Combined Results

  • SketchVLM improves visual reasoning task accuracy by +28.5 points and annotation quality by +48.3% over alternative sketching approaches.The accuracy average covers VPCT, Ball Drop, Maze, and Counting; annotation quality is measured on a 1–5 VLM-judged drawing quality scale.Source: Table C1 · p. 29, Table C2 · p. 29

C.2 Connect-the-Dots … D.1 Ball Drop

These appendix sections report task-specific findings for connect-the-dots, counting, shape drawing, and part labeling, alongside additional-model evaluations and qualitative ball-drop examples. The supplied passages emphasize annotation errors, grounding and recall limitations, and trajectory quality, but provide few quantitative comparisons beyond a 37% error-free connect-the-dots rate.

C.2 Connect-the-Dots

  • Nano Banana Pro completes connect-the-dots without errors only 37% of the time, while Gemini-2.5-Flash-Image often adds nonexistent points or connects dots in the wrong order.The accompanying figure also reports category-wise mean MSE for with-grid and without-grid conditions, but the supplied passage does not give the values.Source: Figure C2 · p. 29, Figure C1 · p. 30

C.3 Counting

  • SketchVLM’s predicted counts align well with target objects according to text-location accuracy, although the supplied passage also notes substantially weaker grounding.The passages do not identify the comparator for the weaker-grounding statement or provide metric values.Source: §C.3 · p. 30

C.4 Drawing Shapes around Objects

  • For drawing shapes around objects, sketch-based localization improves AP50 for medium and large objects but reduces it for small objects; SketchVLM matches precision yet has lower recall because it produces fewer true positives.Different prompt variations do not improve AP performance for this task.Source: Table C4 · p. 30, §C.4 · p. 31, Table C6 · p. 31

C.5 Part Labeling

  • The part-labeling results include an error-type breakdown comparing original models and SketchVLM for labeling and part labeling.The supplied passages provide no breakdown values.Source: Table C7 · p. 31, §C.5 · p. 31

C.6 Additional Model Results

  • Additional-model results report single-turn ablations using sketches and coordinate grids, with RMSE for connect-the-dots and accuracy for other tasks, plus average turns by task group in multi-turn evaluation.Connect-the-dots also includes order accuracy, defined as the percentage of samples with correct point ordering.Source: Table C8 · p. 32, Table C9 · p. 32

D Qualitative Samples

  • Qualitative ball-drop examples show Gemini-3-Pro-Preview producing the most accurate single-turn annotations, while NanoBanana Pro, ThinkMorph, and ViLaSR often cross walls and give wrong answers.This comparison is shown on both VPCT and the paper’s Ball Drop dataset.Source: Figure D1 · p. 33, Figure D2 · p. 34

D.1 Ball Drop

  • On the Ball Drop task, SketchVLM boosts VLM accuracy and enables trajectory paths that closely simulate the ground truth.The supplied figure caption provides no numerical accuracy values.Source: Figure D3 · p. 35

D.2 Maze Navigation … D.6 Part Labeling

The appendix provides qualitative comparisons across maze navigation, connect-the-dots, counting, shape drawing, and part labeling. The supplied passages identify example types and prompts but do not report comparative outcomes for these tasks.

D.2 Maze Navigation

D.3 Connect Dots

D.5 Drawing Shape

  • Figure D10 provides a qualitative comparison on drawing-shape tasks; an accompanying prompt asks models to locate every oven and refrigerator.Source: Figure D10 · p. 42, §D.5 · p. 42

D.6 Part Labeling

  • Figure D11 compares part-labeling examples, while prompts specify object-specific parts to label, including bench arms, backs, legs, seats, stretchers, and tabletops.Source: Figure D11 · p. 43, §D.6 · p. 43

D.7 Connect-the-Dots: Grid versus No Grid … E VLM-Judge Details

The supplementary analyses show that drawing precision and performance vary with coordinate choices, and that Bézier curves and prior-annotation text can improve drawing. Human–VLM ratings align moderately with human judgments, which are more consistent with one another.

D.7 Connect-the-Dots: Grid versus No Grid

  • Appending a reference coordinate grid can make connect-the-dots annotations more precise, but does not help overall performance.Source: Figure D12 · p. 44

D.8 Connect-the-Dots: Bézier Curves versus Lines

  • Using Bézier curves instead of straight lines lets the model connect dots in fewer strokes and draw less jagged, more aesthetically pleasing shapes.Source: Figure D13 · p. 45

D.9 Gemini-3-Pro-Preview Coordinate Systems

  • With Gemini-3-Pro-Preview, a 0–2000 coordinate system often preserves shape but can compress or shift annotations; adaptation usually occurs, though failures reduce performance.The authors hypothesize that the coordinate-system change also explains worse performance with the grid.Source: Figure D14 · p. 46

D.10 Multi-turn Ablation

  • In multi-turn drawing, withholding the text representation of prior annotations degrades quality: the model may redraw earlier strokes and fail to connect new ones to the existing trajectory.Source: Figure D15 · p. 47

E VLM-Judge Details

  • The authors evaluated Gemini-3-Flash-Preview as a judge of annotation quality and annotation–text alignment against independent human ratings using a shared 1–5 rubric.Three independent annotators rated annotations from Ball Drop, VPCT, and Maze Navigation across five models.Source: E VLM-Judge Details · p. 48
  • Human–human agreement was κ = 0.84 ± 0.04 and Pearson r = 0.85 ± 0.04, versus human–VLM agreement of κ = 0.51 ± 0.02 and r = 0.52 ± 0.01.Agreement was measured with quadratically weighted Cohen’s κ and Pearson correlation.Source: E VLM-Judge Details · p. 48, Table E1 · p. 48, Table E2 · p. 49
  • VLM ratings correlate positively with human judgments across tasks and models but can miss subtle errors such as trajectories clipping through walls, so humans remain more reliable for fine-grained evaluation.The VLM judge is used for cost-effective large-scale comparisons.Source: E VLM-Judge Details · p. 48

E.1 Qualitative Examples … F.2 SketchVLM System Prompt

The appendices show qualitative answer and quality-score comparisons, describe the quality rubrics and provider inference settings, and specify task prompts and multi-turn guards for SketchVLM.

E.1 Qualitative Examples

  • The qualitative examples compare original sketching-model and judge answers for valid and invalid paths, and show VLM-Judge quality scores for VPCT and maze paths.The score figures cover VPCT, valid maze paths, and invalid maze paths.Source: Figure E1 · p. 50, Figure E2 · p. 51, Figure E3 · p. 52

E.2 Rubric Prompts

  • The rubrics grade VPCT and ball-drop sketch quality and maze path-navigation quality, with graders instructed to assess only the attributes specified by each rubric.The prompts show the original and annotated image to the grader; the rubric figures are E6 for VPCT and ball drop and E7 for maze navigation.Source: Figure E6 · p. 55, Figure E7 · p. 56

F.1 API Settings

  • Table F1 summarizes model inference settings across providers.Source: Table F1 · p. 57

F.2 SketchVLM System Prompt

  • The counting prompt directs the model to number every object using text-only SVG strokes, with exactly one point near each object’s center.It also specifies readable contrasting colors and text sizing adjusted for object size and image resolution.Source: §F.2 · p. 60
  • The generic labeling prompt restricts labels to the specified parts and requires SVG text strokes rather than curves.It also prohibits inventing additional part names.Source: §F.2 · p. 60
  • The system prompt configures image resolution and coordinate origin, with coordinate examples adjusted for top-left or bottom-left origins.The prompt parameters are res_x, res_y, and origin; examples use x0y{res_y} for the top-left origin and x0y0 for the bottom-left origin.Source: Figure F3 · p. 60, §F.2 · p. 60
  • In multi-turn generation, a one-stroke guard limits drawing turns to one stroke, followed by a final-answer guard after a turn with no strokes.Source: Figure F4 · p. 61

F.3 SketchVLM Output Example · G Other Baselines · G.1 Baselines

The section shows a Gemini-3-Pro-Preview sketch output on VPCT and describes how image-editing and fine-tuned sketching models are evaluated as baselines.

F.3 SketchVLM Output Example

  • Fig. F5 shows a SketchVLM output on VPCT prompted with Gemini-3-Pro-Preview, with each stroke colored differently for viewing.Source: Figure F5 · p. 62

G.1 Baselines

  • For Nano Banana Pro, the baseline workflow uses its generated sketch as input to Gemini-3-Pro-Preview for the final task answer.Source: §G.1 · p. 63
  • For Connect-the-Dots, Nano Banana Pro also receives manual evaluation because its outputs do not always map cleanly to the structured sketch representation.Source: §G.1 · p. 63
  • ViLaSR and ThinkMorph use the same task prompts as SketchVLM, but their sketch-conditioned performance is reported without baseline VQA accuracy.Because these models are trained to always output a sketch, they have no meaningful “no-sketch” baseline mode.Source: §G.1 · p. 63

Related papers

Loading 2604.22875v2…