Source-linked AI summary
Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
Jintao Tong, Jiaqi Gu, Yujing Lou, Lubin Fan, Yixiong Zou, Yue Wu, Jieping Ye, Ruixuan Li
TL;DR
MLLMs often struggle with visual imagination because their reasoning is primarily linguistic, despite visual and textual information sharing a feature space. SkiLa interleaves textual thoughts with continuously generated latent sketch tokens, grounding them through visual-semantic reconstruction. The paper reports superior vision-centric performance and strong generalization across diverse multimodal benchmarks.
Problem
MLLM reasoning is largely restricted to text, limiting visual imagination and performance on vision-intensive tasks.
Method
SkiLa uses a hybrid autoregressive process that alternates textual thoughts with latent sketch tokens trained to reconstruct visual semantics from intermediate sketch images.
Results
SkiLa achieves superior performance on vision-centric tasks and strong generalization across diverse multimodal benchmarks, including +3.9% on MME and +0.9 on POPE.
Takeaways & Limitations
The results support unified multimodal reasoning in which visual cognition and linguistic thoughts are integrated within the model’s latent reasoning process.
Abstract
from arXiv · showhide
While Multimodal Large Language Models (MLLMs) excel at visual understanding tasks through text reasoning, they often fall short in scenarios requiring visual imagination. Unlike current works that take predefined external toolkits or generate images during thinking, however, humans can form flexible visual-text imagination and interactions during thinking without predefined toolkits, where one important reason is that humans construct the visual-text thinking process in a unified space inside the brain. Inspired by this capability, given that current MLLMs already encode visual and text information in the same feature space, we hold that visual tokens can be seamlessly inserted into the reasoning process carried by text tokens, where ideally, all visual imagination processes can be encoded by the latent features. To achieve this goal, we propose Sketch-in-Latents (SkiLa), a novel paradigm for unified multi-modal reasoning that expands the auto-regressive capabilities of MLLMs to natively generate continuous visual embeddings, termed latent sketch tokens, as visual thoughts. During multi-step reasoning, the model dynamically alternates between textual thinking mode for generating textual think tokens and visual sketching mode for generating latent sketch tokens. A latent visual semantics reconstruction mechanism is proposed to ensure these latent sketch tokens are semantically grounded. Extensive experiments demonstrate that SkiLa achieves superior performance on vision-centric tasks while exhibiting strong generalization to diverse general multi-modal benchmarks. Codes will be released at https://github.com/TungChintao/SkiLa.
1. Introduction
MLLM reasoning remains primarily linguistic, limiting visual imagination, so SkiLa enables interleaved textual and latent visual thoughts in a unified reasoning process.
- Text-only reasoning forces MLLMs to verbalize visual problems, limiting performance on tasks requiring rich visual-spatial intuition.The limitation is especially pronounced for tasks such as jigsaw solving and object localization.
- SkiLa expands MLLM autoregression to generate continuous latent sketch tokens as visual thoughts alongside textual thoughts.Its hybrid process dynamically alternates between textual next-token prediction and visual sketching.
- Latent visual semantics reconstruction grounds visual thoughts by training generated latent tokens against embeddings extracted from intermediate sketch images.The sketch module supplies reconstruction targets during training, while the model generates the interleaved reasoning sequence.
- SkiLa is intended to support flexible visual-text reasoning without predefined external actions, including visual imagination and seamless transitions between reasoning modes.The introduction describes this as a unified, multi-step reasoning paradigm inspired by internal visual-text interaction.
- Experiments report strong performance on vision-centric tasks and generalization across diverse multi-modal benchmarks.The paper frames these results as evidence for intrinsic unified reasoning capability.
2. Related Works
Prior multimodal reasoning methods either verbalize visual concepts or rely on predefined tools, while latent approaches generally separate visual reasoning from textual thoughts.
- SkiLa’s training and inference alternate textual thinking with visual sketching, using a training-only sketch module to ground latent sketch tokens in visual semantics.The sketch module extracts embeddings from sketch images as reconstruction targets.
- 2.1. Multi-modal Chain-of-Thought: Multimodal Chain-of-Thought methods extend step-by-step reasoning but remain constrained to linguistic representations of visual concepts.This linguistic restriction hinders performance on vision-intensive tasks.
- Tool-driven approaches provide actions such as cropping, drawing, or marking regions, but depend on predefined toolkits with fixed action spaces.Visual Sketchpad generates code for auxiliary visual aids, while DeepEyes generates boxes and uses zoom-in tools.
- Continuous latent reasoning has been explored for efficiency, and multimodal variants reconstruct auxiliary visual information or image crops in latent space.These approaches move reasoning beyond discrete token space but focus on visual-only reasoning.
- Existing latent multimodal methods do not interleave visual and textual thoughts, reducing interpretability because they provide neither inspectable intermediate images nor corresponding explanations.This separates visual reasoning from the textual reasoning trace.
3. Methods
SkiLa adds latent sketching to a standard MLLM, allowing text and continuous visual tokens to form a unified autoregressive reasoning trace trained with textual prediction and visual reconstruction losses.
- SkiLa combines a vision encoder, modality projector, and LLM backbone, then adds a training-only sketch encoder and projector for visual-semantic targets.The sketch module encodes intermediate sketch images into projected visual embeddings used during training.
- The model switches modes using special sketch-start and sketch-end tokens, generating continuous latent sketch tokens between ordinary textual thoughts.The unified trace contains either text tokens or latent sketch tokens, with special tokens omitted from its compact notation.
- 3.2. Unified Training Objective: Latent sketch tokens are generated autoregressively by feeding each previous hidden state as the next input embedding and matching fixed-length visual targets.The training target is formed by compressing projected sketch features into K salient vectors, such as K = 27.
- 3.2. Unified Training Objective: The visual reconstruction loss uses mean squared error between generated latent tokens and corresponding target visual-semantic vectors.The reconstruction parameters are jointly trained, while discrete tokens use standard next-token prediction with a sketch mask excluding latent tokens.
- 3.2. Unified Training Objective: SkiLa jointly optimizes next-token prediction and latent sketch reconstruction through a weighted objective controlled by λSketch.Textual thoughts, special tokens, and final answers receive cross-entropy supervision.
- At inference, the sketch module is discarded, and latent sketch generation proceeds within the MLLM without additional parameters or computation relative to a standard MLLM.A maximum of 2K latent tokens prevents visual thoughts from overshadowing textual reasoning before standard decoding resumes.
4. Experiments
SkiLa is evaluated across vision-centric, multimodal, ablation, and qualitative settings, with results showing gains from unified visual-text reasoning and latent sketch supervision.
- Vision-Centric and Multimodal Tasks: Visual semantics reconstruction provides larger gains on fine-grained visual tasks such as BLINK Jigsaw Puzzle, where holistic image coherence benefits from visual simulation.The authors contrast this with linguistic descriptions, which they report as insufficient for capturing the full spectrum of visual information.
- Vision-Centric and Multimodal Tasks: SkiLa consistently outperforms Qwen2.5-VL 7B across vision-centric perception, high-resolution understanding, and multimodal reasoning benchmarks, with unified reasoning achieving the best overall performance.The comparison includes vision-centric and multimodal multi-choice tasks, and the reported results also surpass GPT-4o and Claude3.7-Sonnet.
- General Tasks: SkiLa generalizes beyond its primary reasoning tasks, improving TextVQA by +3.3 for SkiLa-V and +1.8 for SkiLa, MME by +3.9%, and POPE by +0.9.These gains are reported relative to the Direct SFT baseline and correspond to stronger text-grounded reasoning, perceptual and cognitive reasoning, and factuality.
- Ablation Studies: The default SigLIP2 sketch encoder achieves the highest scores in almost every benchmark among the compared encoders, motivating its selection over QwenViT and CLIP.The comparison uses 28 QwenViT, 24 CLIP, and 27 SigLIP2 sketch tokens.
- Ablation Studies: Performance improves as reconstructed sketch tokens increase from 1 to 27 but drops slightly at 81, so 27 tokens are selected to balance visual and textual thoughts.The authors also set the latent sketch reconstruction loss weight to 0.5; with 27 tokens, stronger supervision is important, whereas performance with 9 tokens is relatively robust to loss-weight changes.
- Discussion: Qualitative cases show spatial imagination, hallucination mitigation, and adaptive unified reasoning through visual simulation, implicit zooming, and switching between modalities.SkiLa mentally simulates 3D objects from 2D patterns, identifies fine-grained visual details, and adapts its reasoning process to the task.
5. Conclusion
SkiLa introduces unified reasoning for MLLMs by interleaving textual thoughts with latent visual sketch tokens, grounded through latent visual semantics reconstruction. Experiments show superior vision-centric performance and strong generalization across multimodal benchmarks.
- SkiLa alternates textual thinking with visual sketching to generate interleaved multimodal reasoning traces, while latent visual semantics reconstruction grounds the visual thoughts.
- SkiLa achieves superior performance on vision-centric tasks and strong generalization to diverse general multimodal benchmarks.
6. Training Data Details
The training data is filtered from Zebra-CoT to retain samples suitable for learning interleaved textual thoughts and visual sketch targets. The resulting dataset contains 101K samples, with detailed statistics reported in Table 4.
- The training set contains 101K filtered samples from Zebra-CoT, excluding 3D data and overly complex intermediate sketches.Zebra-CoT contains 182K interleaved text-image reasoning samples before filtering.
- The selected samples are reformatted as interleaved sequences of textual thoughts and visual sketch targets for SkiLa training.
7. Choice of Loss Function
The study compares cosine similarity and mean squared error for latent sketch reconstruction. MSE performs better because latent visual representations encode semantics through both direction and magnitude.
- MSE consistently yields higher performance than cosine similarity for latent sketch reconstruction.Cosine similarity captures directional alignment, whereas MSE minimizes reconstruction error and accounts for directional and magnitude information.
8. More Cases
Additional cases illustrate SkiLa’s visual reasoning capabilities across fine-grained grounding, latent scene completion, and cross-image visual alignment. These examples show how unified latent reasoning supports visual refinement and manipulation without relying on external tools.
- The supplementary cases are presented as broader examples across additional task types.
- Fine-Grained Visual Grounding: SkiLa internally focuses on key image regions to recover subtle visual details without external region-based tools.The case identifies that all three glasses contain water while avoiding explicit region-operation latency.
- Latent Scene Completion: SkiLa reconstructs missing content in partially occluded images and selects semantically consistent completions for jigsaw-style tasks.The case contrasts with Vision-R1, which incorrectly identifies the missing part when forming a coherent global scene.
- Cross-Image Visual Alignment: SkiLa performs unified reasoning over fine-grained visual semantics for cross-image visual alignment.