PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, Ranjay Krishna

arXiv:2406.09403v3cs.CVcs.CL

TL;DR

Multimodal LMs lack the visual intermediate steps people use to support reasoning. Visual SKETCHPAD gives them tools to plan, generate, and inspect sketches during reasoning, and it reports consistent improvements across mathematical and vision tasks, while requiring more computing resources than direct language generation.

  • Problem

    Multimodal LMs lack the visual reasoning scaffold needed for tasks where people use intermediate sketches to simplify symbolic and spatial reasoning.

  • Method

    Visual SKETCHPAD lets multimodal LMs plan and reason through intermediate visual artifacts created with drawing programs and specialist vision models.

  • Results

    SKETCHPAD consistently improves performance across mathematical and computer-vision tasks and sets new state-of-the-art results across the evaluated tasks.

  • Takeaways & Limitations

    The framework enables multimodal LMs to use visual artifacts such as auxiliary lines, plots, graphs, bounding boxes, masks, crops, and depth estimates during reasoning.

  • Takeaways & Limitations

    SKETCHPAD requires more computing resources than directly outputting language tokens and is evaluated primarily with existing off-the-shelf LMs.

Abstract

from arXiv · show

Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.

1 Introduction

Visual Sketchpad addresses the absence of intermediate visual reasoning actions in multimodal LMs by letting them create and analyze sketches. It applies this framework across mathematical and visual reasoning tasks, with reported improvements over models without sketching.

  • Motivation: Sketching supports human communication, ideation, and problem-solving by conveying visuo-spatial and abstract relationships directly.Examples include geometry support lines, engineering prototypes, architectural blueprints, and maps.
  • Motivation: Current multimodal LMs face benchmarks requiring symbolic grounding and spatial understanding where intermediate sketches could enhance reasoning.Relevant tasks include geometry diagrams, complex mathematics, object detection, depth estimation, BLINK, and V*Bench.
  • Framework: Visual SKETCHPAD lets multimodal LMs produce visual artifacts as part of mixed textual, programmatic, and visual reasoning.For geometry, the model can add an auxiliary line and annotated angles to the original diagram.
  • Framework: SKETCHPAD combines sketching with specialist vision models, such as depth estimation, to improve visual reasoning.For stacked cookies, an intermediate depth estimate reveals overlapping objects at different depths.
  • Results: 11.2% average gain in GPT-4o performance was reported across four mathematical task categories, while computer-vision performance improved across seven task types.Reported vision improvements include 14.3% on V*Bench, 12.1% on BLINK depth, and 9.7% on BLINK semantic correspondence.

2 Related Work

Visual SKETCHPAD extends multimodal tool-use, visual prompting, and LM-agent paradigms by allowing models to choose and create visual artifacts during reasoning. Its distinguishing feature is visual sketching as an intermediate reasoning step rather than text alone or a predefined visual plan.

  • Visual programming and tool-use: Prior visual programming and tool-use methods decompose complex vision tasks into substeps executed through specialized vision tools.Visprog and ViperGPT use LMs to generate Python code that sequentially invokes such tools.
  • Visual prompting: Visual prompting augments natural images with labels, masks, or other visual prompts to improve multimodal grounding.SKETCHPAD generalizes this direction by letting LMs decide which visual prompting to use during multimodal reasoning.
  • Positioning: SKETCHPAD combines visual sketching with multimodal agent behavior, extending existing visual prompting and tool-use approaches.The framework accepts multimodal inputs and outputs and allows visual artifacts to participate in the reasoning process.
  • LMs as agents: LM-agent research treats language models as systems that can both reason and act across domains including software engineering, robotics, vision, and GUI navigation.SKETCHPAD differs by creating visual artifacts to facilitate reasoning, whereas prior LM agents primarily generate text during reasoning.

3 Visual SKETCHPAD

Visual SKETCHPAD implements iterative multimodal reasoning in which an LM plans a sketch, executes a programmatic action, observes the resulting artifact, and then answers. The framework uses plotting packages and specialist vision models to generate task-dependent visual sketches.

  • 3.1 Overview of SKETCHPAD: Given a multimodal query, SKETCHPAD alternates among Thought, Action, and Observation before producing a final response.The thought is a sketching plan, the action synthesizes a program, and the observation is the resulting visual artifact.
  • 3.1 Overview of SKETCHPAD: At each time step, the model analyzes context to generate a plan for the next action, such as drawing an auxiliary line parallel to a geometry edge.The context contains the query and previous thoughts, actions, and observations.
  • 3.1 Overview of SKETCHPAD: The model executes each plan by generating Python code that manipulates visual and textual content, and the environment returns an updated observation.The interaction continues until the model issues a Terminate action and provides the answer.
  • 3.1 Overview of SKETCHPAD: Unlike primarily text-based agent frameworks, SKETCHPAD supports multimodal observations and actions so models can reason with sketches they created.This design allows visual artifacts to become part of subsequent planning and reasoning.
  • 3.2 Sketching via Code Generation: Sketching is implemented through programs that call Python plotting packages or specialist vision models.Matplotlib and networkx support mathematical plots, while detection, segmentation, and marking tools create boxes, masks, and labels for vision tasks.

4 Sketching to Solve Math Problems

The paper evaluates SKETCHPAD on geometry, functions, graph algorithms, and game strategies, using visual artifacts to support reasoning across mathematical representations. Results report consistent gains over base models, including large improvements on graph problems and mathematical-function tasks.

  • Tasks: SKETCHPAD is evaluated on four mathematical task categories: geometry, mathematical functions, graph algorithms, and game strategies.The evaluation uses Geometry3K and IsoBench tasks alongside visual and textual mathematical inputs.
  • Sketching methods: Auxiliary lines, function plots, graph visualizations, and chess boards provide task-specific representations for reasoning.For example, networkx converts an adjacency matrix into a graph structure, while textual chess moves become visual board states.
  • Results: 66.3% accuracy on maximum flow was achieved by GPT-4o with SKETCHPAD, a 41.3% improvement over its base model.The paper also reports over 90% GPT-4 Turbo accuracy and over 88% GPT-4o accuracy on convexity and parity tasks.

5 Sketching to Solve Computer Vision Tasks

SKETCHPAD augments multimodal LMs with specialist vision tools that produce visual artifacts for complex visual reasoning. It improves performance across all evaluated tasks and establishes new state-of-the-art results.

  • 5 Sketching to Solve Computer Vision Tasks: SKETCHPAD enhances multimodal LMs’ visual reasoning and establishes new SOTAs on all 7 evaluated tasks.The tasks include V*Bench, MMVP, BLINK, relative depth, spatial reasoning, jigsaw, visual correspondence, and semantic correspondence.
  • 5.1 Vision Specialists as Sketching Tools in SKETCHPAD: Vision specialists provide callable modules for detection, segmentation, depth estimation, sliding-window search, zoom-in, cropping, and image overlay.These modules generate bounding boxes, masks, depth maps, image patches, or manipulated images that the LM can use during reasoning.
  • 5.2 Results: GPT-4o with SKETCHPAD sets new state-of-the-art results across all complex visual reasoning tasks.Table 2 compares SKETCHPAD with GPT-4 Turbo, GPT-4o, and other multimodal LM baselines.
  • 5.2 Results: 14.3% accuracy improvement is achieved by GPT-4o on V*Bench, surpassing the previous state of the art SEAL.GPT-4 Turbo improves by 18.5% on V*Bench; SKETCHPAD also yields average BLINK gains of 6.6% for GPT-4 Turbo and 9.0% for GPT-4o.
  • 5.2 Results: Vision-specialist use is task-dependent, with detection, search, and zoom-in used for V*Bench, depth estimation for relative depth, and detection plus segmentation for spatial reasoning.GPT-4o uses specialists more often than GPT-4 Turbo; on semantic correspondence, it uses segmentation for 40% of instances versus less than 20% detection use by GPT-4 Turbo.
  • 5.2 Results: SKETCHPAD is the only compared augmentation framework that consistently improves performance on all single-image tasks.SoM can hurt performance on some tasks, while Visprog performs worse than the base LM across all tasks; composed vision-module errors may propagate.

6 Analysis and Discussion

The analysis attributes SKETCHPAD’s effectiveness to visual artifacts complementing language and enabling adaptive planning. Human-study and open-source-model experiments provide supporting evidence for its reasoning utility and transferability.

  • Visual artifacts complement language by representing dense information such as depth and segmentation directly.
  • SKETCHPAD lets multimodal LMs plan from their own intermediate artifacts and revise plans after specialist errors.
  • On geometry, humans draw the same auxiliary line as GPT-4o 80% of the time, while human judges rate GPT-4o plans valid in 92.8% of vision instances.
  • Most errors arise from vision-specialist failures and simple visual question-answering mistakes rather than planning.
  • Oracle SKETCHPAD visual artifacts from GPT-4o produce consistent improvements on open-source LLaVA-NEXT models’ math reasoning.

7 Conclusion

The paper presents Visual SKETCHPAD as a framework for multimodal LMs to create and reason over intermediate sketches. It reports improvements across tasks while identifying computational cost and reliance on off-the-shelf models as limitations or future directions.

  • Visual SKETCHPAD equips multimodal LMs to generate intermediate sketches for reasoning.
  • For mathematics, it visualizes auxiliary lines, functions, graphs, and games; for vision, specialists provide artifacts such as boxes and masks.
  • SKETCHPAD enhances performance across all evaluated tasks and sets new state-of-the-art results.
  • The framework requires more computing resources than directly outputting language tokens.
  • The study uses existing off-the-shelf LMs, leaving training integration and broader applications for future work.

B More Qualitative Examples

The qualitative appendix shows how SKETCHPAD prompts combine thoughts, actions, code, and visual tools across vision and mathematical tasks. Examples include iterative detection, zooming, plotting, and graph reasoning.

  • Prompts for computer vision tasks: Vision prompts instruct the model to reason in THOUGHT and ACTION steps, using Python tools and terminating with an answer.
  • Prompts for computer vision tasks: The balloon example illustrates recovery from failed detection by switching to sliding-window detection and inspecting image patches.
  • Prompts for computer vision tasks: The model answers that the red balloon is below the white balloon after locating both balloons in patches.
  • Prompts for math tasks: Math prompts remove computer-vision specialists and add example plotting code for tasks such as parity, convexity, and graph analysis.
  • Prompts for math tasks: Graph prompts cover connectivity, graph isomorphism, and maxflow using adjacency matrices or visual graph representations.

D Dataset Statistics

The appendix reports dataset statistics for the benchmarks used in the study, including IsoBench and multiple vision-task datasets.

  • Tables 5 and 6 summarize statistics for IsoBench, BLINK, MMVP, and V*Bench.
  • Table 5 presents IsoBench dataset statistics.
  • Table 6 presents statistics for the vision-task datasets.

E Costs

The paper identifies the cost of running SKETCHPAD on each task as a dedicated experimental consideration, documented in Table 7.

  • The paper directs readers to Table 7 for GPT-4o task-level running costs.
  • The supplied passages identify task-level cost reporting but provide no numerical cost values.
  • Table 7 reports the cost of running SKETCHPAD on each task.

F Impact Statement

The impact statement presents SKETCHPAD as a framework for human interaction with multimodal language models and notes both interpretability benefits and potential misuse risks.

  • SKETCHPAD proposes a new way for humans to interact with language models through language and sketches.
  • The framework may make language models more interpretable by eliciting reasoning with both language and sketches.
  • If misused, language models may generate harmful vision and text artifacts, motivating further research on multimodal safety.

NeurIPS Paper Checklist

The checklist records that the paper’s contributions and scope are reflected in its abstract and introduction, while experimental and societal-impact information is documented across the submission. It also states that the paper makes no theoretical claims and discusses limitations in Section 7.

  • Claims and scope: The abstract and introduction are stated to reflect the paper’s contributions and scope.
  • Limitations: The paper discusses limitations in Section 7 and identifies assumptions and scope as relevant considerations for assessing empirical claims.
  • Theory assumptions and proofs: The paper makes no theoretical claims, so theorem assumptions and proofs are not applicable.
  • Experimental reproducibility: Implementation details, test settings, compute resources, and statistical-score calculation procedures are reported for reproducibility.
  • Open access to data and code: The dataset is accessible online, and the authors state that they are committed to releasing the code.
  • Societal impact: The impact discussion addresses potential negative societal effects, including harms from intended use, incorrect results, or misuse, and mentions mitigation strategies as relevant considerations.
Loading 2406.09403v3…