PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, Ranjay Krishna

arXiv:2406.09403v3cs.CVcs.CL

TL;DR

Multimodal LMs are increasingly expected to solve tasks where people use intermediate sketches to simplify reasoning. Sketchpad gives them tools to draw and reason over visual artifacts. It consistently improves base-model performance across tasks, with GPT-4o averaging an 11.2% improvement on math tasks.

  • Problem

    Multimodal LMs are increasingly expected to solve tasks where people draw intermediate sketches to simplify reasoning.

    Source: §1 · p. 1

  • Method

    Sketchpad enables multimodal LMs to plan, create visual sketches, and reason further by analyzing those artifacts.

    Source: §3 · p. 3

  • Results

    Sketchpad consistently improves base-model performance across tasks, with GPT-4o averaging an 11.2% improvement on math tasks.

    Source: §4.1 · p. 6, §5.2 · p. 7

  • Takeaways & Limitations

    The findings support visual sketching as a useful aid for multimodal reasoning across mathematical and visual tasks.

    Source: §7 · p. 9

    Sketchpad requires more computing resources than directly outputting language tokens.

    Source: §7 · p. 9

Abstract

from arXiv · show

Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.

1 Introduction

Visual SKETCHPAD gives multimodal LMs a sketchpad for visual artifacts within mixed textual, programmatic, and visual reasoning. It addresses the lack of sketch-based reasoning scaffolds for tasks where visual sketches can support spatial understanding and problem-solving.

Motivation

  • Sketches convey visuo-spatial ideas directly and support communication, ideation, and problem-solving, but current multimodal LMs lack a scaffold for sketch-based reasoning.Geometry3K [32], IsoBench, BLINK [9], and V*Bench [51] include tasks where intermediate sketches could aid reasoning.Source: §1 · pp. 1–2

Framework

  • SKETCHPAD prompts visual LMs to create visual artifacts as part of mixed textual, programmatic, and visual reasoning, such as adding auxiliary lines and angle annotations to geometry diagrams.For visual reasoning, it can also use intermediate depth estimates to identify overlapping objects, such as stacked cookies.Source: §1 · p. 2

Evaluation scope

  • The evaluation spans geometry, functions, graph algorithms, and strategy games, plus computer-vision tasks in depth, spatial reasoning, jigsaw, visual and semantic correspondence, MMVP, and V*Bench [9] [44] [51].Sketches include plotted functions from language-only inputs; visual tools include segmentation masks, crops, bounding boxes, zooming, and overlays.Source: §1 · pp. 2–3

Results

  • GPT-4o with SKETCHPAD improves V*Bench by 14.3%, BLINK depth by 12.1%, and BLINK semantic correspondence by 9.7%, setting new state-of-the-art results across tasks.Across four math task categories, SKETCHPAD consistently improves baseline GPT-4o performance, with an average gain of 11.2%; it also consistently improves performance across seven computer-vision task types.Source: §1 · p. 3

Plan analysis

  • SKETCHPAD’s generated plans align well with human-created plans and exhibit similar reasoning patterns.The authors present this alignment as evidence of the framework’s effectiveness and as an opening for more capable and interpretable multimodal intelligence.Source: §1 · p. 3

2 Related Work

The related-work section situates Sketchpad among multimodal tool-use, visual prompting, and LM-agent research. It presents Sketchpad as a framework that lets LMs use visual artifacts during reasoning.

Visual programming and tool-use

  • Unlike Visprog [14] and ViperGPT [38], Sketchpad lets LMs revise plans based on intermediate visual artifacts, improving performance and robustness on complex multimodal tasks.Visprog and ViperGPT generate Python code to invoke specialized vision tools according to a predefined LM plan.Source: §2 · p. 3

Visual prompting

  • Sketchpad generalizes visual prompting by letting LMs choose prompts, such as labeled segmentation masks, as part of multimodal reasoning.SoM [55] shows that labeled segmentation masks can enhance GPT-4V’s visual grounding; related work also studies visual prompts in 3D [25] and robotics [34].Source: §2 · p. 3

LMs as agents

  • Unlike prior LM agents that generate only text while reasoning, Sketchpad creates visual artifacts to facilitate reasoning over multimodal inputs and outputs.LM agents have been studied in software engineering [20] [61], robotics [34], vision [29] [57], and GUI navigation [54] [23] [53].Source: §2 · p. 3

3 Visual SKETCHPAD

SKETCHPAD lets multimodal LMs plan and reason through visual sketches, using an iterative thought–action–observation process and code-generated drawing tools. The framework works without finetuning and can combine general-purpose plotting with specialist vision models.

3 Visual SKETCHPAD

  • SKETCHPAD treats sketches as intermediate reasoning steps: the model plans a sketch, creates it, analyzes the result, and uses it to answer the query.The framework requires no finetuning or training; multimodal LMs can be prompted to sketch with it out of the box.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Overview of SKETCHPAD

  • At each turn, SKETCHPAD plans from multimodal context, executes a visual or textual action, observes the updated context, and repeats until terminating with an answer, unlike text-focused interaction [58].Actions can modify visual and textual content, allowing the model to plan and reason with its sketches.Source: §3.1 · pp. 3–4

3.2 Sketching via Code Generation

  • SKETCHPAD sketches through generated Python programs that call task-specific tools, including plotting packages and specialist vision models.For math, it uses matplotlib and networkx; for vision, detection tools draw bounding boxes, while segmentation and marking tools add masks and numbered labels. The model receives tool signatures and docstrings, executes generated code, and can display the resulting sketch in its next observation.Source: §3.2 · pp. 4–5

4 Sketching to Solve Math Problems

Sketchpad applies visual sketching to geometry, functions, graph algorithms, and chess, improving multimodal LMs’ performance across these math tasks. The section describes drawing workflows and reports gains over base models and other baselines.

4 Sketching to Solve Math Problems

  • For Geometry3K [32], Sketchpad edits matplotlib code to add auxiliary lines; for IsoBench functions, it plots curves for visual inspection, addressing LMs’ errors on f(−x) and derivatives.The function tasks are parity classification and identifying convexity or concavity.Source: §4 · p. 5
  • On IsoBench graph problems —connectivity, maximum flow, and graph isomorphism—Sketchpad uses NetworkX to draw graphs from adjacency matrices for visual reasoning.Source: §4 · p. 5
  • For IsoBench chess winner identification, Sketchpad draws boards from FEN using Python’s chess library, turning textual move notations into visual positions; game-strategy gains are 3% ∼10%.Source: §4 · p. 5, §4.1 · p. 6

4.1 Results

  • 11.2% for GPT-4o and 23.4% for GPT-4 Turbo are the average improvements from Sketchpad, which consistently improves base-model performance across all tasks.Table 1 reports accuracy on geometry, graph algorithms, mathematical functions, and games, and its caption states that Sketchpad outperforms all baselines on most tasks.Source: §4.1 · p. 6, Table 1 · p. 6
  • 66.3% accuracy on maximum flow is a 41.3% improvement for GPT-4o, while GPT-4 Turbo exceeds 90% and GPT-4o exceeds 88% on convexity and parity classification.The passage also reports game-strategy gains of 3% ∼10%.Source: §4.1 · p. 6

5 Sketching to Solve Computer Vision Tasks

Section 5 evaluates SKETCHPAD on complex visual reasoning tasks, using specialist vision models as sketching tools. It reports improved visual reasoning and new state-of-the-art results across seven tasks.

5.1 Vision Specialists as Sketching Tools in SKETCHPAD

  • SKETCHPAD exposes callable Python tools for detection boxes and coordinates, numbered segmentation masks, and depth maps, using Grounding-DINO [30], SoM [55] models, and DepthAnything [56].The segmentation models are SegmentAnything and Semantic-SAM [24].Source: §5.1 · pp. 6–7
  • Other tools search images with a 16-window sliding grid for queried items, crop within a bounding box, and overlay two images using supplied alpha values.The search window is one-third of the image size, with a step size of two-ninths.Source: §5.1 · p. 7

5.2 Results

  • On BLINK tasks, SKETCHPAD yields average 6.6% absolute accuracy gain for GPT-4 Turbo and 9.0% for GPT-4o, and improves multi-image jigsaw, visual correspondence, and semantic correspondence tasks.Source: §5.2 · p. 7
  • 12.1% accuracy improvement for GPT-4o on relative depth, compared with 2.4% for GPT-4 Turbo; the authors attribute GPT-4o’s greater benefit to better understanding of generated depth maps.Source: §5.2 · p. 7
  • 18.5% accuracy improvement for GPT-4 Turbo and 14.3% for GPT-4o on V*Bench; GPT-4o also set new SOTA on all tasks.Table 2 reports accuracy on complex visual reasoning tasks.Source: §5.2 · p. 7, Table 2 · p. 8
  • Tool use was task-dependent: both models used detection, search, and zoom for V*Bench, depth estimation for relative depth, and detection and segmentation for spatial reasoning; GPT-4o used more tools.For semantic correspondence, GPT-4o used segmentation on 40% of instances, while GPT-4 Turbo used detection less than 20% of the time and rarely used segmentation; performance was 58.3% versus 42.4%, respectively.Source: §5.2 · pp. 7–8, Figure 4 · p. 8
  • SKETCHPAD was the only compared framework to improve consistently across single-image tasks; SoM [55] helped spatial reasoning but could hurt other tasks, while Visprog [14] underperformed the base LM on all tasks.The Visprog comparison replaced its LM and VQA modules with the corresponding GPT-4 model; the authors note module errors may propagate when composed by a program [21] [18].Source: §5.2 · pp. 8–9, Table 3 · p. 8

6 Analysis and Discussion

The discussion attributes Sketchpad’s effectiveness to visual artifacts that complement language and let models adapt their plans, and reports human-plan agreement and consistent math-task improvements for open-source LLaVA models using GPT-4o’s sketches.

Why does Sketchpad work?

  • Sketchpad offers visual representations for dense information [9] and lets models adapt plans after inspecting intermediate artifacts, unlike predefined-plan modular vision methods [14] [38] [55].The discussion also notes that model plans resemble human plans and may benefit from training data with similar reasoning patterns.Source: §6 · p. 9

Do LMs have the same plans as humans?

  • 80% of geometry problems had humans draw GPT-4o’s same auxiliary line, while two subjects rated its vision-task plans valid in 92.8% of instances.Most errors arose from vision-specialist failures, such as missed objects, or mistakes in simple visual question answering rather than planning.Source: §6 · p. 9

Experiments on open-source models

  • Oracle Sketchpad using GPT-4o’s final visual artifacts consistently improves open-source LLaVA-NEXT models’ math-task performance and mathematical reasoning.The experiments use the visual artifacts from GPT-4o + Sketchpad’s last action as image inputs.Source: §6 · p. 9, Table 4 · p. 9

7 Conclusion

Visual SKETCHPAD equips multimodal LMs with sketching tools and vision specialists to reason over visual artifacts, improving performance across tasks and achieving state-of-the-art results. The conclusion notes higher computing costs and points to training natively multimodal models and expanding applications such as robotics.

7 Conclusion

  • SKETCHPAD improves LM performance across all tasks and sets new state-of-the-art results by sketching during mathematical reasoning and using vision specialists for visual reasoning.Specialists can provide visualizations such as object-detection bounding boxes and segmentation masks for further planning and reasoning.Source: §7 · p. 9
  • SKETCHPAD requires more computing resources than directly outputting language tokens; future work may explore instruction-tuning natively multimodal models such as Unified-IO 2 [31] and Chameleon [41], which can output both text and images.Source: §7 · p. 9
  • SKETCHPAD could extend to robotics, for example by highlighting small objects in crowded spaces, zooming the camera, or using depth estimation to aid navigation.Source: §7 · p. 9

B More Qualitative Examples … Thought + Action 1

The appendix shows qualitative examples and a computer-vision prompt setup, then traces a balloon-position question where the model switches to sliding-window search after object detection fails to annotate the balloons.

B More Qualitative Examples

C Prompts

  • The same prompt template is used for computer-vision tasks; code is displayed differently for visualization but is input to the model as text.Source: C Prompts · p. 17

SYSTEM PROMPT

  • The system prompt directs the model to produce a THOUGHT each turn and, once it has the answer, respond with ANSWER and terminate.Source: SYSTEM PROMPT · p. 17

Initial Prompt + Request

  • The initial prompt supplies tools for detection, sliding-window search, segmentation, depth, zooming, and overlays, and asks the model to plan actions using visual outputs.Source: Initial Prompt + Request · pp. 17–19

Thought + Action 0

Observation 0

  • The detection execution succeeds, and the model is prompted to generate the next thought and action.Source: Observation 0 · p. 20

Thought + Action 1

  • Because detection did not annotate the balloons, the next thought chooses sliding-window search and runs it over the image.Source: Thought + Action 1 · p. 20

Observation 1

  • The sliding-window execution succeeds and presents Patch 1; the model is then prompted for another thought and action.Source: Observation 1 · pp. 20–21

Thought + Action 2

This section shows a visual reasoning trace that answers a relative-position question without another action, alongside task-specific math prompts for visual reasoning.

Thought + Action 2

  • The trace concludes that the red balloon is below the white balloon and selects answer (A), with no further action needed.The conclusion is based on seeing both balloons in patches 4 and 5.Source: Thought + Action 2 · p. 21
  • Math-task prompts remove computer-vision specialists and add example plotting code while retaining a similar format to the vision-task prompts.The parity, convexity, graph connectivity, and graph isomorphism prompt captions specify code to generate images, following a similar format to [8].Source: Thought + Action 2 · p. 22, Figure 6 · p. 22, Figure 7 · p. 22
  • The graph maxflow prompt asks models to solve the maxflow problem, while the Winner ID prompt asks them to analyze the game outcome.Source: Figure 10 · p. 25, Figure 11 · p. 25

D Dataset Statistics

The appendix reports statistics for datasets used in the paper, covering IsoBench, BLINK, MMVP, and V*Bench.

D Dataset Statistics

E Costs · F Impact Statement

The section points to per-task GPT-4o running costs and describes Sketchpad’s potential research, interaction, and interpretability benefits alongside possible misuse and an open multimodal safety concern.

E Costs

F Impact Statement

NeurIPS Paper Checklist

The checklist says the abstract and introduction reflect the paper’s contributions and scope, limitations are discussed, and the work makes no theoretical claims; it also reports reproducibility and ethics-related disclosures.

Claims, limitations, and theory

  • The checklist says the abstract and introduction match the paper’s contributions and scope, limitations are discussed in section 7, and the paper makes no theoretical claims.Source: NeurIPS Paper Checklist · pp. 27–28

Reproducibility and experimental details

  • The authors report that implementation details, test settings, evaluation-score calculations, and compute-resource requirements are provided in sections 3, C, D, 4, 5, and E.Source: NeurIPS Paper Checklist · pp. 28–30

Open access to data and code

  • The checklist answers No on open access to code and data: the dataset is online, and the authors commit to releasing the code.Source: NeurIPS Paper Checklist · p. 29

Ethics and societal impacts

Release safeguards and research subjects

  • The authors say no data or models are being released because the work concerns inference of language models, and mark new-asset documentation and human-subjects research as not applicable.Source: NeurIPS Paper Checklist · pp. 31–32

Licenses for existing assets

  • The checklist answers Yes on crediting existing assets and respecting their terms, noting that codebases are acknowledged and dataset and model URLs are provided.Source: NeurIPS Paper Checklist · p. 31

Related papers

Loading 2406.09403v3…