PaperlayerView exact revision on arXiv

Source-linked AI summary

SketchVLM: Vision language models can annotate images to explain thoughts and guide users

Brandon Collins, Logan Bolton, Hung Huy Nguyen, Mohammad Reza Taesiri, Trung Bui, Anh Totti Nguyen

arXiv:2604.22875v2cs.CVcs.AI

TL;DR

Modern VLMs typically answer image questions with text that can be difficult to verify. SketchVLM instead generates editable, non-destructive SVG overlays, outperforming sketching models by +28.5 percentage points in accuracy and +48.3% in annotation quality.

  • Problem

    Modern VLMs typically respond to image questions with text, which can be difficult for users to verify.

  • Method

    SketchVLM prompts VLMs to produce structured strokes converted into editable, non-destructive SVG overlays grounded on the original image without task-specific training.

  • Results

    +28.5 percentage points in accuracy and +48.3% in annotation quality are achieved relative to sketching models across the evaluated tasks.

  • Takeaways & Limitations

    These visual explanations let users verify model reasoning at a glance, unlike text-only responses and image-editing baselines.

  • Takeaways & Limitations

    SketchVLM does not perform well on small VLMs that struggle with instruction following, such as Qwen2.5-VL-7B [3].

Abstract

from arXiv · show

When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.

1 Introduction

SketchVLM addresses the difficulty of verifying text-only VLM answers by generating editable SVG annotations in a separate overlay layer grounded on the original image. Across seven drawing and visual-reasoning tasks, it improves generalizability, accuracy, and annotation quality while preserving the source image and supporting both single- and multi-turn interaction.

  • Motivation and approach: SketchVLM generates SVG annotations in a separate overlay layer grounded on the original image, without modifying it or requiring task-specific training.The framework is evaluated with multiple VLM backbones, including Gemini-3-Pro-Preview [30].
  • Main findings: SketchVLMs draw more accurate ball-drop trajectories, connect dots more accurately, and produce more plausible maze-navigation paths, while Nano Banana often alters the image and produces implausible paths.Specialist VLMs fine-tuned for sketching often fail to generalize to new tasks, whereas SketchVLMs show superior generalizability, accuracy, and annotation quality.
  • Main findings: Across seven tasks—three drawing tasks and four visual-reasoning tasks—SketchVLMs produce more generalizable, accurate, and higher-quality annotations than specialized fine-tuned sketching models.The evaluation covers connecting-the-dots, object-part labeling, shape drawing, physics understanding, counting, and maze navigation.
  • Interaction and guidance: Single-turn and multi-turn SketchVLMs achieve similar accuracy, but single-turn generation is significantly faster; adding an external x–y coordinate grid improves drawing and question-answering accuracy but is unnecessary.These findings identify speed and coordinate guidance as practical trade-offs in deployment.

2 Related Work … 6. VPCT

SketchVLM equips existing vision-language models with editable SVG overlays through visual prompting, structured stroke generation, and XML-to-SVG rendering. The paper evaluates this framework across visual reasoning and drawing tasks, including connection, counting, localization, part labeling, maze navigation, and visual physics prediction.

  • 2 Related Work: SketchVLM enables state-of-the-art VLMs to annotate input images with non-destructive SVG overlays, unlike image-editing models and native multimodal models lacking editable aligned annotations.The framework is positioned against native image-editing, interleaved text–image, and fine-tuned sketching approaches.
  • 3 SketchVLM: The framework combines visual prompting, structured stroke outputs, and XML-to-SVG conversion to render annotations over the source image.A coordinate grid supports spatial precision; two-point strokes become lines, while longer strokes are fit with smooth cubic Bézier curves.
  • 1. Connect-the-Dots: Connect-the-dots evaluation requires models to locate dots and connect them in order across randomly generated patterns, silhouette-derived puzzles, and worksheet-style images.The benchmark contains 100 images across three subsets.
  • 2. Counting Objects: Counting evaluation uses 746 images with object counts from 0 to 10, requiring models to count target objects and place numbered markers on each.The images are drawn from CountBench and Pixmo-Count.
  • 3. Drawing Shapes around Objects: Drawing-shape, part-labeling, and maze benchmarks test object localization, text placement at object parts, and path validity in grid mazes.The localization set contains 1,000 COCO images; part labeling covers 985 images and 52 object classes; mazes vary shortest paths from 3 to 8 steps and include invalid perturbed paths.
  • 4. Part Labeling: Part labeling requires placing correct text labels at corresponding part locations, while maze navigation requires tracing proposed paths and determining whether they reach goals without crossing walls.These tasks use curated part annotations and generated 3 × 3 grid mazes.
  • 6. VPCT: VPCT evaluates whether models can predict the landing container of a dropped ball from 100 hand-crafted images, with single-turn and multi-turn annotation generation illustrated on the same sample.In multi-turn generation, later calls reuse prior annotations in rendered-image and text form.

7. Ball Drop

The Ball Drop benchmark tests whether SketchVLM can predict landing containers while tracing ball trajectories, comparing single-turn and multi-turn annotation against existing visual-annotation approaches. SketchVLM maintains competitive accuracy while producing visual reasoning traces.

  • Benchmark: The benchmark contains 198 synthetic images with one, two, or three randomly placed lines and randomized ball positions, requiring a landing-container choice and trajectory trace.Ground-truth trajectories are generated using PHYRE [4], with harder-to-guess answers and four possible containers.
  • Evaluation: SketchVLM is evaluated in single-turn mode and multi-turn mode, where multi-turn generation produces one stroke per turn using previously rendered annotations and their text representations.The model gives its final text answer on the last turn in the multi-turn setting.
  • Metrics: Performance is assessed through task accuracy and annotation–text alignment, measuring both answer correctness and whether visual traces faithfully support the text answer.The evaluation distinguishes informative annotations from plausible but contradictory ones.
  • Results: SketchVLM produces visual reasoning traces while maintaining competitive accuracy on the Ball Drop task.Table 2 reports this overall comparison, while the evaluation also considers annotation–text alignment rather than answer correctness alone.

5 Results

SketchVLM achieves strong task accuracy and produces annotations that align closely with its answers, enabling visual verification across reasoning and drawing tasks. Results also show benefits from grid prompting and single-turn generation, while revealing limitations for small objects and annotation-quality evaluation.

  • Ablations: Adding both the grid and sketching prompt performs best overall, but the grid raises Connect-the-Dots RMSE from 5.92 to 99.34.Consequently, the reported configuration uses the grid except for Connect-the-Dots, where results are better without it.
  • Drawing tasks: SketchVLM locates up to 35 connecting-the-dots points with 5.92 RMSE and orders points correctly 74% and 99% of the time.The method maintains spatial accuracy and logical coherence across many strokes, while other models often connect points out of order.
  • Object detection: Stroke-based rectangles improve medium-object performance by +0.6 and large-object performance by +1.4 but reduce small-object detection by -10.1, lowering AP50 from 63.1 to 58.8.The analysis attributes the degradation to lower recall, while precision matches the original model.
  • Visual reasoning: 96.0% VPCT and 79.7% Ball Drop accuracy demonstrate SketchVLM’s strong performance on visual reasoning tasks.The reported SketchVLM results substantially exceed the near-random baseline accuracies described for comparison models.
  • Alignment and annotation quality: 95.5% and 94.2% mean alignment scores show that SketchVLM annotations closely match the model’s text answers.The paper identifies this alignment as enabling users to verify whether the model’s reasoning makes sense.
  • Generation setting: Single-turn generation reaches comparable or higher accuracy than multi-turn generation across all tasks while using about 5.92x fewer turns.This supports single-turn generation as an efficient setting for producing high-quality annotations.

6 Conclusion

SketchVLM is a training-free framework for producing editable, non-destructive SVG annotations that help users verify VLM reasoning visually. It outperforms sketching models in accuracy, transfers to strong open-source VLMs, but struggles with smaller models and may benefit from editable strokes.

  • Conclusion: SketchVLM outperforms sketching models by +28.5 percentage points in accuracy.The result is reported in Tab. C1.
  • Limitations and Future Work: SketchVLM transfers to strong open-source VLMs like Kimi K2.5 [44], but performs poorly on smaller models such as Qwen2.5-VL-7B [3].The limitation is attributed to weaker instruction following in smaller VLMs.
  • Limitations and Future Work: Allowing models to undo and edit strokes is a potential way to improve multi-turn performance.This is proposed as future work.

7 Author contribution statement

The authors divided responsibilities across benchmark creation and curation, experiments, method development, evaluation, writing, demo and website development, and project supervision. BC, LB, and HN were major contributors to datasets and experiments.

  • 7 Author contribution statement: BC, LB, and HN created or curated benchmark datasets and ran experiments.HN curated three benchmarks; BC created Connect-the-Dots; LB created Ball Drop and Maze Navigation.
  • 7 Author contribution statement: HN led evaluation and the human-versus-VLM agreement study, while BC led method development, code, ablations, and evaluations on Connect-the-Dots and VPCT.LB led Ball Drop and Maze Navigation evaluation and ran open-source model, annotation-quality, and annotation-text alignment experiments.
  • 7 Author contribution statement: BC and LB led manuscript writing, all authors edited and reviewed it, and BC, LB, and AN developed the demo.BC led website development with additional contributions from LB.
  • 7 Author contribution statement: BC and LB served as technical team leads, while AN supervised the project.AN also contributed to demo development with BC and LB.

A Real World Applications … C.1 Combined Results

The paper demonstrates SketchVLM in real-world instructional and multi-turn applications, evaluates it across diverse datasets and tasks, and reports improved combined visual-reasoning accuracy and annotation quality over alternative sketching approaches.

  • A Real World Applications: SketchVLM visually guides users through setting up an AWS EC2 instance and supports multi-turn explanations for PyTorch containers and GitHub CI/CD pipelines.It responds with visual annotations, whereas ChatGPT responds with text only, and it supports a variety of real-world use cases.
  • B.1 Connect-the-Dots: The connect-the-dots dataset contains 100 images spanning random-dot, outline, and clutter-varying subsets.Random-dot images contain 4–10 numbered dots, while outline puzzles are derived from 30 Openclipart SVG silhouettes.
  • B.2 Counting: The counting dataset combines CountBench [5], TallyQA, and Pixmo-Count, totaling 1,189 samples with object counts ranging from 0 to 10.CountBench and TallyQA contribute 746 samples, while Pixmo-Count contributes 443 after filtering.
  • B.3 Drawing Shapes around Objects: The drawing-shapes dataset contains 1,000 selected COCO validation images balanced across object counts, classes, and object sizes.The selection covers small, medium, and large objects.
  • B.4 Part Labeling: The part-labeling dataset selects single-object images from PACO and Pascal-Part with sufficiently large, richly annotated, and class-balanced objects, yielding 985 images across 52 classes.Each object occupies at least 10% of the image and has at least four annotated parts.
  • B.5 Maze Navigation: Maze navigation uses 200 unique 3x3 grids in which models determine whether a commanded path reaches its goal without crossing border walls.Shortest ground-truth paths range from 3 to 8 steps, and one direction is randomly altered to create an invalid path.
  • B.6 Ball Drop: Ball Drop evaluation uses 100 VPCT images to predict the bucket reached by a dropped ball and adds generated benchmarks for evaluating trajectory drawing.VPCT provides physics-understanding cases but no ground-truth ball trajectories.
  • C.1 Combined Results: +28.5 points in average visual-reasoning accuracy over alternative sketching approaches, and +48.3% in VLM-judged annotation quality on a 1–5 scale.The accuracy average covers VPCT, Ball Drop, Maze, and Counting tasks; the quality comparison uses the combined annotation results.

C.2 Connect-the-Dots … D.1 Ball Drop

Across connect-the-dots, counting, shape drawing, part labeling, additional-model, and ball-drop evaluations, SketchVLM reveals task-specific gains and limitations in accuracy, grounding, recall, and annotation fidelity. Qualitative results further show that Gemini-3-Pro-Preview produces the most accurate ball-drop annotations, while SketchVLM improves trajectory simulation and accuracy.

  • C.2 Connect-the-Dots: Nano Banana Pro completes connect-the-dots without errors in 37% of cases, outperforming Gemini-2.5-Flash-Image, which often adds nonexistent points or misorders dots.Mean MSE is also reported with and without grids, with negative percentage change indicating improvement.
  • C.2 Connect-the-Dots: In connect-the-dots, Gemini-2.5-Flash-Image frequently introduces nonexistent points or violates dot order, while Nano Banana Pro performs significantly better but remains imperfect.
  • C.3 Counting: SketchVLM achieves high text-location accuracy for counting, indicating that predicted counts align well with target objects, but its grounding is substantially weaker.
  • C.4 Drawing Shapes around Objects: Sketch-based localization improves AP50 for medium and large objects but reduces performance on small objects, while SketchVLM preserves precision but lowers recall through fewer true positives.Different prompt variations do not improve AP performance for the drawing-shape task.
  • C.5 Part Labeling: The part-labeling evaluation reports error-type breakdowns for original models and SketchVLM across labeling and part-labeling tasks.
  • C.6 Additional Model Results: Additional-model ablations compare strokes, system prompts, and coordinate grids using RMSE for connect-the-dots, accuracy for other tasks, and order accuracy for point ordering.Multi-turn evaluation also reports average turns per task group.
  • D Qualitative Samples: Qualitative ball-drop examples show Gemini-3-Pro-Preview producing the most accurate annotations, whereas NanoBanana Pro, ThinkMorph, and ViLaSR often cross walls and answer incorrectly.
  • D.1 Ball Drop: SketchVLM boosts VLM accuracy on ball drop while producing trajectory paths that closely simulate the ground-truth data.

D.2 Maze Navigation … D.6 Part Labeling

The appendix provides qualitative comparisons across maze navigation, connect-the-dots, counting, shape drawing, and part-labeling tasks. These examples visualize model outputs, overlays, and—in connect-the-dots—MSE where reported.

  • D.2 Maze Navigation: Maze-navigation examples show model outputs on both valid and invalid paths.The appendix presents separate qualitative examples for successful and unsuccessful path predictions.
  • D.3 Connect Dots: Connect-the-dots results are qualitatively compared across multiple models, GPT-5 reasoning levels, multi-turn variants, and sketching baselines.Comparisons cover random dots, worksheets, and outlines; the random-dot figure displays overlays and MSE, with Gemini-2.5-Pro reported at Grid MSE 1080 and Qwen3-235B as N/A.
  • D.4 Counting: Counting-task examples qualitatively compare the models’ visual outputs.The supplied figure caption identifies a qualitative comparison on counting tasks but does not provide task-specific outcome values.
  • D.5 Drawing Shape: Drawing-shape examples compare Nano Banana, Gemini 3.0 Pro, and SketchVLM overlays for locating specified object categories.One shown prompt asks models to locate every oven and refrigerator, while the figure compares shape-drawing outputs.
  • D.6 Part Labeling: The part-labeling demonstrations span increasingly detailed object vocabularies, including anatomical and structural components.Listed labels range from bench parts such as arm and seat to bird and person parts including eyes, limbs, torso, and facial features.
  • D.6 Part Labeling: Part-labeling examples compare Nano Banana, Gemini 3.0 Pro, and SketchVLM on labeling specified parts of diverse objects.The examples cover benches, spoons, bottles, buses, birds, and people, with prompts restricting labels to listed parts.

D.7 Connect-the-Dots: Grid versus No Grid … E VLM-Judge Details

The supplementary analyses examine coordinate grids, curve representations, coordinate systems, and multi-turn state handling for connect-the-dots annotations, then validate the VLM judge against human ratings. The judge agrees moderately with humans and is useful for scalable comparison, but human evaluation remains more reliable for subtle errors.

  • D.7 Connect-the-Dots: Grid versus No Grid: Adding a reference coordinate grid makes annotations more precise but does not improve performance.The grid is appended to the input image edge.
  • D.8 Connect-the-Dots: Bézier Curves versus Lines: Bézier curves connect dots in fewer strokes and produce less jagged, more aesthetically pleasing shapes than straight lines.The comparison concerns annotation efficiency and visual smoothness rather than task accuracy.
  • D.9 Gemini-3-Pro-Preview Coordinate Systems: Using a 0–2000 coordinate system often preserves overall shape but can compress or shift annotations relative to the usual 0–1000 system.Although the model usually adapts, these failures reduce performance and may explain worse results with the grid.
  • D.10 Multi-turn Ablation: In multi-turn generation, omitting the text representation of prior annotations degrades drawing quality because the model redraws earlier strokes instead of connecting new ones.This failure prevents proper integration with the existing trajectory.
  • E VLM-Judge Details: Human–VLM agreement is moderate (κ = 0.51 ± 0.02; Pearson r = 0.52 ± 0.01), supporting scalable evaluation while missing some subtle logical errors.Ratings are positively correlated across tasks and models, but trajectories clipping through walls can be overlooked.
  • E VLM-Judge Details: SketchVLMs achieve higher mean human annotation-quality scores than other annotation models across the evaluated tasks.Table E4 reports task-level and overall means with standard deviations.

E.1 Qualitative Examples … F.2 SketchVLM System Prompt

The appendices provide qualitative comparisons, grading rubrics, API settings, and system prompts governing SketchVLM’s single-turn and multi-turn annotations. The prompts specify coordinate conventions, structured SVG outputs, task-specific drawing constraints, and guards for multi-turn interaction.

  • E.1 Qualitative Examples: Qualitative examples compare the original sketching model’s answers with judge answers for valid and invalid paths, alongside VLM-Judge scores for VPCT and maze navigation.The comparisons cover both valid and invalid maze paths, while scores are reported separately for VPCT and maze-navigation cases.
  • E.2 Rubric Prompts: The VPCT and ball-drop rubric grades AI annotations against specified attributes using the original and annotated images.The appendix includes both the rubric description and its displayed figure.
  • E.2 Rubric Prompts: The maze-navigation rubric similarly evaluates sketch quality from the original and AI-annotated images using only the provided rubric attributes.The rubric is documented in text and shown in a dedicated figure.
  • F.1 API Settings: Model inference settings across providers are summarized in a dedicated API-settings table.The supplied passage identifies this as Table F1 but does not provide its cell values.
  • F.2 SketchVLM System Prompt: During multi-turn generation, a one-stroke guard limits drawing turns, followed by a final-answer guard after a turn produces no additional strokes.The system prompt figures document the shared prompt used for single-turn and multi-turn SketchVLM models.
  • F.2 SketchVLM System Prompt: Counting prompts require one numbered SVG text stroke per object, exactly one point per stroke, and a complete wrapper even when no objects are present.The prompt also requires outputs to follow a specified XML-like answer and strokes structure.
  • F.2 SketchVLM System Prompt: Generic labeling prompts restrict annotations to listed parts and require one SVG text stroke for each label in an exact XML-like structure.The label example places text at an xAyB point and assigns an identifier such as label_head.
  • F.2 SketchVLM System Prompt: The system prompt supports configurable x/y resolutions and either a top-left or bottom-left coordinate origin, with examples adapted to the selected origin.The coordinate parameters are res_x, res_y, and origin; res_x and res_y determine coordinate counts, while origin selects the corner.

F.3 SketchVLM Output Example · G Other Baselines · G.1 Baselines

The appendix illustrates SketchVLM’s VPCT sketch output and defines image-editing and fine-tuned sketching baselines, including evaluation choices necessitated by their output formats and training. Fine-tuned models are assessed only in the sketch-conditioned setting because they lack a meaningful no-sketch mode.

  • F.3 SketchVLM Output Example: SketchVLM produces colored, editable-style stroke annotations for VPCT when prompted with Gemini-3-Pro-Preview.The example colors each stroke differently solely to improve visual inspection.
  • G Other Baselines: The appendix separates illustrative SketchVLM outputs from baseline procedures that require either image editing or models trained to sketch unconditionally.These distinctions determine how each baseline is evaluated.
  • G Other Baselines: The baseline suite includes image-editing models that first generate a sketch and then pass the edited image to Gemini-3-Pro-Preview for the final task answer.Nano Banana Pro is evaluated using this two-stage procedure.
  • G.1 Baselines: Nano Banana Pro receives an additional manual Connect-the-Dots evaluation because its outputs do not always map cleanly to the structured sketch representation.This manual evaluation supplements the model’s benchmark assessment for that task.
  • G.1 Baselines: Fine-tuned sketching models such as ViLaSR and ThinkMorph use the same task prompts as SketchVLM.These models are trained to always output a sketch.
  • G.1 Baselines: Because fine-tuned sketching models always produce sketches, their main results omit baseline VQA accuracy and report only sketch-conditioned performance.They therefore lack a meaningful no-sketch baseline mode.
Loading 2604.22875v2…