PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs

Huanyu Zhang, Wenshan Wu, Chengzu Li, Ning Shang, Yan Xia, Yangyu Huang, Yifan Zhang, Li Dong, Zhang Zhang, Liang Wang, Tieniu Tan, Furu Wei

arXiv:2510.24514v1cs.CVcs.CL

TL;DR

MLLMs perform well on visual understanding but struggle with complex reasoning requiring visual planning and imagination. Latent Sketchpad adds autoregressive visual latents through a Context-Aware Vision Head and renders them with a pretrained Sketch Decoder, achieving comparable or superior reasoning while producing interpretable traces. The framework is designed as a modular extension applicable across diverse MLLM backbones.

  • Problem

    MLLMs struggle with complex multimodal reasoning involving precise spatial reasoning and dynamic visual grounding, despite strong visual understanding.

  • Method

    Latent Sketchpad interleaves visual-latent generation with textual reasoning through a Context-Aware Vision Head and uses a standalone Sketch Decoder to render interpretable sketches.

  • Results

    The framework yields interpretable visual traces while retaining reasoning performance and supporting plug-and-play use across diverse pretrained MLLMs.

  • Takeaways & Limitations

    Latent Sketchpad extends frontier MLLMs with visual thinking without relying on external tools, enabling broader visual reasoning and human inspection of internal traces.

  • Takeaways & Limitations

    The approach demands substantially more data for generalization.

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) excel at visual understanding, they often struggle in complex scenarios that require visual planning and imagination. Inspired by how humans use sketching as a form of visual thinking to develop and communicate ideas, we introduce Latent Sketchpad, a framework that equips MLLMs with an internal visual scratchpad. The internal visual representations of MLLMs have traditionally been confined to perceptual understanding. We repurpose them to support generative visual thought without compromising reasoning ability. Building on frontier MLLMs, our approach integrates visual generation directly into their native autoregressive reasoning process. It allows the model to interleave textual reasoning with the generation of visual latents. These latents guide the internal thought process and can be translated into sketch images for interpretability. To realize this, we introduce two components: a Context-Aware Vision Head autoregressively produces visual representations, and a pretrained Sketch Decoder renders these into human-interpretable images. We evaluate the framework on our new dataset MazePlanning. Experiments across various MLLMs show that Latent Sketchpad delivers comparable or even superior reasoning performance to their backbone. It further generalizes across distinct frontier MLLMs, including Gemma3 and Qwen2.5-VL. By extending model's textual reasoning to visual thinking, our framework opens new opportunities for richer human-computer interaction and broader applications. More details and resources are available on our project page: https://latent-sketchpad.github.io/.

1 Introduction

Latent Sketchpad extends MLLMs with internal visual thoughts to address complex multimodal reasoning that language-only and external-tool approaches struggle to handle. It interleaves visual-latent generation with textual reasoning and provides sketch visualizations for interpretation.

  • 1 Introduction: Current MLLMs remain challenged by complex multimodal reasoning involving precise spatial reasoning and dynamic visual grounding.
  • 1 Introduction: Human-inspired visual sketches provide complementary structure for simulating scenarios, testing alternatives, and refining plans.
  • 1 Introduction: External visual tools are constrained by predefined capabilities and environments, while pixel-oriented generators prioritize image realism over reasoning-oriented abstractions.
  • 1 Introduction: Latent Sketchpad interleaves autoregressive text generation with visual latents, extending MLLMs beyond language-only deliberation.The latents remain in representation space during reasoning rather than being decoded into images.
  • 1 Introduction: A Context-Aware Vision Head generates coherent visual latents from current and previous representations, while a standalone Sketch Decoder renders them as interpretable sketches.
  • 1 Introduction: Evaluations report interpretable visual traces, retained reasoning performance, plug-and-play modularity, and applicability across diverse pretrained MLLMs.

2 Latent Sketchpad

Latent Sketchpad extends pretrained MLLMs with visual latents interleaved with text, using contextual generation for visual coherence and a decoder for interpretable sketches.

  • 2 Latent Sketchpad: Latent Sketchpad interleaves autoregressive text generation with continuous visual latents, preserving the backbone’s reasoning process while adding visual thought.The Vision Head is trained separately while MLLM parameters remain frozen.
  • 2.2 Context-Aware Vision Head: The Context-Aware Vision Head generates each visual latent from global preceding-image context and local current-image context, supporting long-range memory and within-image coherence.Generated latents are projected back into the language embedding space so autoregressive generation can continue.
  • 2.2 Context-Aware Vision Head: Visual generation proceeds token by token after a start-of-image signal, ends after a fixed number of visual tokens and an end-of-image token, then resumes text generation.The Vision Head applies causal attention so each image token depends on prior context.
  • 2.2 Context-Aware Vision Head: The Vision Head is supervised by latent regression against pretrained vision-encoder features using similarity or distance criteria such as cosine similarity or L1 distance.The target latent comes from pretrained visual features rather than pixel-space supervision.
  • 2.3 Pretrained Sketch Decoder: The standalone Sketch Decoder maps vision-encoder features into pretrained VAE latent space and decodes them into human-interpretable sketch images.An AlignerNet performs the feature-space mapping before a frozen VAE decoder produces pixels.
  • 2.3 Pretrained Sketch Decoder: The Sketch Decoder is trained with reconstruction, latent-distribution, and embedding losses while freezing the vision encoder and VAE, using Quick, Draw! sketches for pretraining.The dataset contains 50 million sketch-style images across 345 categories.

3 Experiments

Experiments evaluate Latent Sketchpad on MAZEPLANNING with Gemma3, Qwen2.5-VL, proprietary models, and frozen-backbone plug-and-play configurations. Results indicate strong cross-model applicability, preserved reasoning performance, and improved complex spatial reasoning when visual traces are available.

  • 3 Experiments: Evaluation uses MAZEPLANNING action sequences, with Success Rate measuring complete correct solutions and Progress Rate measuring consecutively correct actions before the first mistake.The dataset contains 47.8K training mazes and a 500-maze test set spanning easy and hard sizes.
  • 3.2 Experimental Results: Strong proprietary models achieve less than 20% success rate and below 50% progress rate on MAZEPLANNING, highlighting the task’s difficulty.The reported failures primarily stem from difficulty tracking evolving spatial states.
  • 3.2 Experimental Results: Latent Sketchpad integrated with GPT-4o yields significant gains in success and progress rates, reaching performance comparable to dedicated reasoning models and surpassing o1 on progress rate.The generated visual traces provide complementary spatial cues that guide reasoning.
  • 3.2 Experimental Results: Latent Sketchpad can attach to MLLMs without altering backbone parameters or noticeably degrading their original reasoning performance.The Vision Head is trained independently with the backbone frozen, supporting plug-and-play integration.
  • 3.2 Experimental Results: Latent Sketchpad adapts across Gemma3 and Qwen2.5-VL, externalizing internal visual features as explicit reasoning traces while extending multimodal reasoning capacity.The experiments support applicability across architecturally different pretrained backbones.

4 Discussion and Analysis

Latent Sketchpad produces structurally stable visual traces, generalizes across vision encoders, and improves maze reasoning while retaining advantages across maze sizes and model configurations.

  • Visualization Quality: The Sketch Decoder achieves consistently high SSIM on unseen MAZEPLANNING samples across OpenCLIP, Qwen2.5-VL, and Gemma3 encoders.The encoders use substantially different pretraining schemes, supporting decoder compatibility across encoder types.
  • Qualitative Analysis: Rendered traces may have lower perceptual quality for arrows or digits, but they maintain structural stability during planning.The paper attributes this stability to semantic context guiding the visual trajectory through the Context-Aware Vision Head.
  • Quantitative Analysis: 75.6% VSR with Gemma3 Latent Sketchpad exceeds the 70% baseline SR, while generated visual traces raise Table 1 performance from 70% to 72.2%.A consistent improvement trend is also reported for Qwen2.5-VL.
  • Quantitative Analysis: High LCR across both models reflects preserved spatial structure, which correlates with successful path planning measured by VSR.LCR measures maze layout preservation, while VSR measures whether a valid path is drawn in the correct layout.
  • Further Analysis: Latent Sketchpad maintains higher success rates than baselines across maze scales, although performance declines as maze size increases.Larger mazes introduce greater spatial complexity and make accurate planning more challenging.
  • Ablations: Freezing the connector severely impairs spatial understanding, whereas connector adaptation is critical during downstream fine-tuning.The reported failure includes confusing left and right directions, with similar trends on Qwen2.5-VL.
  • Ablations: Reconstructing intermediate visual thoughts with pixel-level perturbations improves visual-thought accuracy and task success rates.The augmentation preserves spatial semantics while adding appearance variability.
  • Ablations: L1 regression loss consistently outperforms cosine similarity across evaluation metrics, better preserving latent spatial and semantic fidelity.The comparison evaluates alternative objectives for training the Vision Head.

5 Related Work

Related work enhances multimodal reasoning through textual prompting, external visual tools, native visual thoughts, latent reasoning, and unified multimodal generation.

  • Multimodal Reasoning: CoT prompting and external tools such as cropping and zooming provide finer-grained visual perception during multimodal reasoning.These approaches augment reasoning with textual intermediate steps or targeted visual inputs.
  • Multimodal Reasoning: MVoT and Visual Planning demonstrate that models can generate visual thoughts natively for step-by-step reasoning.This line of work moves beyond tool-assisted visual inputs toward internally generated visual intermediates.
  • Latent Reasoning: Latent reasoning performs multi-step inference directly in continuous hidden states instead of generating explicit reasoning tokens.The related-work discussion contrasts this with the expressiveness constraints of natural-language reasoning.
  • Unified Multimodal Generation: Unified multimodal generation extends autoregressive modeling with image and other output modalities, often using diffusion-based decoders.These models typically combine autoregressive modeling with modality-specific generation components.

6 Conclusion

Latent Sketchpad equips pretrained MLLMs with internal visual thoughts, renders them as interpretable sketches, and evaluates the approach on a new maze-planning dataset.

  • Conclusion: Latent Sketchpad adds a Context-Aware Vision Head that generates internal visual representations within the autoregressive reasoning loop.The framework enables visual features to function as internal visual thoughts alongside text.
  • Conclusion: A separately pretrained Sketch Decoder translates latent visual representations into interpretable sketches without relying on external tools.The sketches support human understanding and interaction with the model’s reasoning traces.
  • Conclusion: The authors report that Latent Sketchpad extends frontier MLLMs with interpretable visual traces and enriched reasoning capabilities.The conclusion describes these results as coming from extensive experiments.
  • Dataset: The MAZEPLANNING dataset supports multimodal step-wise inference in complex environments.It is constructed specifically to facilitate research on visual-language reasoning.

A.1 Dataset Overview

MAZEPLANNING provides multimodal maze trajectories with decision-point actions and state-based visual validation, including in-distribution and larger out-of-distribution tests.

  • Dataset Overview: MAZEPLANNING contains 47.8K training mazes, plus 500 in-distribution and 200 larger 6×6 out-of-distribution test mazes.The OOD set is designed to assess generalization to more complex scenarios.
  • Dataset Overview: Each maze is annotated with a multimodal trajectory intertwining visual and textual reasoning steps.Actions are defined at decision points rather than every grid movement to reflect flexible human-like reasoning.
  • Dataset Overview: The trajectory uses three abstract actions: go forward, turn left, and turn right.Go forward continues until the next decision point, while turns rotate before moving forward.
  • Dataset Overview: Reasoning is segmented into states of k ∈[4, 6] actions, followed by rendered-path generation and validity checking.Inference proceeds to the next state when the current state is judged valid and coherent.

A.2 Dataset Curation

MazePlanning uses synthetically curated mazes with unique solution paths, action annotations, and GPT-4o-generated textual reasoning narratives aligned to those paths.

  • Dataset Construction: MazePlanning mazes were manually constructed with varying grid sizes and branching patterns while guaranteeing a single unique solution path.Each maze was designed to remain unambiguously solvable.
  • Annotations: Ground-truth trajectories were converted into manually annotated actions at key decision points, including forward movement and turns.
  • Dataset Examples: The dataset includes textual reasoning examples and action sequences paired with visual maze states.
  • Reasoning Narratives: GPT-4o synthesized coherent reasoning narratives from ground-truth action sequences to couple spatial reasoning, language, and task context.

B Implementation Detail

The implementation combines transformer-based visual components with masked multimodal fine-tuning, latent reconstruction augmentation, and model-specific training configurations.

  • Architecture: The Context-Aware Vision Head uses 2 cross-attention and 8 self-attention layers, while the Sketch Decoder uses 12 encoder and 12 decoder layers.
  • Training: Training masks non-initial images with probability 0.5 to expose Gemma3 and Qwen2.5-VL to mixed text-only and multimodal reasoning trajectories.
  • Inference: During multimodal inference, <start_of_image> is inserted after </actions> to trigger generation of the subsequent visual state.
  • Augmentation: Latent Reconstruction Augmentation repeatedly re-encodes and decodes intermediate visual thoughts for up to k ∈[0, 3] rounds while preserving semantic content.

B.5 Implementation details of GPT-4o + Latent Sketchpad.

The GPT-4o integration uses a plug-and-play loop that feeds rendered visual states back into reasoning, while modality alignment improves semantic coherence but can leave maze layouts unstable.

  • GPT-4o Integration: The plug-and-play interface invokes Latent Sketchpad after correct preceding actions and feeds the rendered maze state back to GPT-4o for iterative refinement.
  • Modality Alignment: Unfreezing Liquid’s image embedding layer substantially improves semantic coherence compared with frozen alignment, producing more structured and plausible images.
  • Modality Alignment: Aligned Liquid may alter the underlying maze layout, making visually clear solutions incorrect because they no longer correspond to the original maze.
  • Failure Analysis: Proprietary reasoning models can lose spatial location during multi-step maze reasoning, causing compounding path-prediction errors.

C.2.2 Latent Sketchpad

Latent Sketchpad is evaluated through qualitative error analysis and system comparisons, revealing strong baseline improvements alongside distinct in-distribution and out-of-distribution failure modes.

  • In-Distribution Failures: Latent Sketchpad occasionally produces in-distribution paths that cut through walls or teleport, yielding incorrect plans despite locally coherent actions.
  • Out-of-Distribution Failures: On larger unseen mazes, Gemma3’s sketches gradually degrade, while Qwen2.5-VL struggles to preserve layouts and generate valid paths.
  • Failure Analysis: The analysis identifies structural violations in familiar settings and cumulative degradation in novel environments as two distinct failure types.
  • Baseline Improvement: Text-only fine-tuning substantially improves the limited performance of the original Gemma3 model on MAZEPLANNING.
  • Evaluation Scope: Qwen2.5-VL results are omitted in this setting because weaker instruction following prevented consistent, meaningful outputs.

C.4 Task Performance

Across MAZEPLANNING configurations, Latent Sketchpad’s benefit depends on the backbone, while its decoder reconstructs sketches that preserve reasoning-relevant structure.

  • Connector tuning strongly affected performance: freezing it produced poor results for both backbones, while one epoch substantially improved Qwen2.5-VL.The reported convergence behaviors differed between Qwen2.5-VL and Gemma3.
  • 82.6 visual success for Qwen2.5-VL with Latent Sketchpad was comparable to its 82.4 text-only baseline, whereas Gemma3 reached 75.6 versus 70.The comparison uses the two-epoch connector-unfrozen setting.
  • Reconstructed sketches recovered key geometric and semantic patterns across unseen structural layouts, despite minor degradation in fine-grained lines and color fidelity.The authors consider the current reconstruction quality sufficient for visual reasoning and identify fidelity improvement as future work.
  • Decoded visual thoughts organized spatial information and guided step-by-step decisions, capturing sufficient structural cues for accurate multimodal reasoning without photorealistic detail.These visualizations were produced on MAZEPLANNING tasks using the pretrained Sketch Decoder.
Loading 2510.24514v1…