PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers

Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, Yi R. Fung

arXiv:2506.23918v3cs.CV

TL;DR

Text-centric multimodal reasoning treats vision as static context, leaving a semantic gap between rich visual information and discrete language-based thought. This survey introduces “Thinking with Images,” organizes it into three stages of increasing cognitive autonomy, and synthesizes its methods, evaluations, applications, challenges, and future directions.

  • Problem

    Textual Chain-of-Thought treats vision as a static initial context and reasoning as exclusively textual, creating a semantic gap that flattens visual structure.

  • Method

    The survey organizes “Thinking with Images” into three stages: external tool orchestration, programmatic visual manipulation, and intrinsic imagination or simulation.

  • Results

    The survey synthesizes a field-wide shift from passive visual inputs to dynamic cognitive workspaces, with methods combining visual understanding, explicit scratchpads, and interleaved multimodal reasoning.

  • Takeaways & Limitations

    “Thinking with Images” provides a roadmap toward multimodal systems that use visual information as intermediate reasoning steps for deeper perceptual analysis and physical simulation.

  • Takeaways & Limitations

    Visual reasoning remains computationally expensive because dense visual patches and multi-step intermediate images create a compounding token and computation burden.

Abstract

from arXiv · show

Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-centric approach, however, treats vision as a static, initial context, creating a fundamental "semantic gap" between rich perceptual data and discrete symbolic thought. Human cognition often transcends language, utilizing vision as a dynamic mental sketchpad. A similar evolution is now unfolding in AI, marking a fundamental paradigm shift from models that merely think about images to those that can truly think with images. This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process, transforming vision from a passive input into a dynamic, manipulable cognitive workspace. In this survey, we chart this evolution of intelligence along a trajectory of increasing cognitive autonomy, which unfolds across three key stages: from external tool exploration, through programmatic manipulation, to intrinsic imagination. To structure this rapidly evolving field, our survey makes four key contributions. (1) We establish the foundational principles of the think with image paradigm and its three-stage framework. (2) We provide a comprehensive review of the core methods that characterize each stage of this roadmap. (3) We analyze the critical landscape of evaluation benchmarks and transformative applications. (4) We identify significant challenges and outline promising future directions. By providing this structured overview, we aim to offer a clear roadmap for future research towards more powerful and human-aligned multimodal AI.

1 Introduction

The survey frames “Thinking with Images” as a shift from static, text-only reasoning over visual inputs to dynamic visual cognition, organized across three stages of increasing autonomy. It reviews the paradigm’s foundations, methods, evaluations, applications, challenges, and future directions.

  • The Paradigm Shift: “Thinking about Images” encodes vision once as static context, leaving language as the exclusive medium of reasoning and creating a semantic gap with rich visual information.This bottleneck limits deeper iterative engagement in physical reasoning, spatial manipulation, and long-horizon planning.
  • The Paradigm Shift: “Thinking with Images” treats visual representations as manipulable and verifiable intermediate thoughts, allowing models to actively see, modify, and generate visual information.The paradigm turns vision into a dynamic cognitive workspace rather than a passive input.
  • Three-Stage Framework: The proposed framework charts increasing cognitive autonomy from external tool orchestration, through programmatic visual operations, to intrinsic visual imagination and simulation.The stages distinguish how models achieve visual reasoning goals, progressing from commander to visual programmer to visual thinker.
  • Survey Scope: The survey systematically addresses foundational principles, stage-specific methodologies, evaluation benchmarks, implementation frameworks, applications, challenges, and future directions.Its taxonomy is illustrated through core methodologies, supporting benchmarks and frameworks, and key applications.
  • Survey Position: The survey positions itself as a systematic review of mechanisms that transform images from static premises into dynamic, manipulable cognitive workspaces.It contrasts prior reviews that primarily treat vision as context while language remains the main reasoning vehicle.

2 Foundations of the Thinking with Images Paradigm

“Thinking with Images” treats visual information as an operable intermediate step rather than a fixed input, enabling active exploration, structured reasoning, and generative planning. The paradigm progresses from external tool orchestration through programmatic manipulation to intrinsic visual imagination, while facing major costs in computation, robustness, architecture, and generalization.

  • Core Capabilities: Active visual exploration lets models iteratively query images through operations such as cropping or highlighting to inspect details and object interactions.This capability moves beyond a single holistic interpretation of an image.
  • Core Capabilities: Structured visual reasoning externalizes thought by rendering elements such as auxiliary geometry lines, making abstract relationships perceptually evident.The generated image can transform symbolic deduction into visual pattern recognition and make the reasoning chain more interpretable.
  • Three-Stage Framework: The roadmap advances from fixed-tool orchestration, through code-generated custom analysis, to intrinsic image generation and self-critique.The three stages represent increasing cognitive autonomy, with intrinsic imagination forming a closed visual reasoning loop.
  • Formal Definition: “Thinking with Images” uses visual representations as manipulable and verifiable intermediate reasoning steps instead of treating images as static initial context.Its reasoning history can contain textual or visual steps generated from evolving multimodal context.
  • Unique Challenges: Visual reasoning is computationally expensive because dense image patches make multi-step visual deliberation increasingly prohibitive.The resulting token explosion limits the length and complexity of reasoning paths.
  • Unique Challenges: Visual errors can establish a false perceptual ground truth that supports coherent but irrelevant downstream reasoning.An incorrect zoom-in may redirect the entire analysis toward an unrelated object.
  • Unique Challenges: A single visual strategy cannot cover all tasks: geometry, detail diagnosis, and maze navigation require constructive, analytical, and simulative approaches.Learning a meta-policy to select among diverse strategies remains an unsolved challenge.

3 Stage 1: Tool-Driven Visual Exploration

Stage 1 shifts multimodal reasoning from passive visual input to active inquiry by having models orchestrate predefined visual tools. The stage progresses from prompt-based coordination through supervised and reinforcement learning toward dynamic perception and future skill internalization, while remaining bounded by its fixed toolkit.

  • Tool-Driven Visual Exploration: Stage 1 makes the model a planner that actively analyzes images by orchestrating a predefined suite of external visual tools.Its core transition is from passive perception to active, tool-driven inquiry.
  • Tool Categories: The fixed toolkit spans semantic grounding, perception exploration, and visual reasoning-aid tools that support distinct cognitive purposes.Semantic grounding converts visual content into symbolic text, while perception exploration enables finer inspection of image details.
  • Formulation: At each reasoning step, the model selects a tool and arguments, receives modified images, data layers, or structured text, and integrates the output into its next state.The tool execution engine applies the command to the image and returns the result for continued reasoning.
  • Implementation Approaches: Tool orchestration is developed through prompt-based coordination, supervised fine-tuning for procedural competence, and reinforcement learning for policy optimization.Prompting enables coordination without parameter updates, while training-based approaches cultivate learned visual attention and tool-use behavior.
  • Conclusion and Future Frontiers: The paradigm is practical, transparent, and debuggable, but models cannot invent visual operations beyond the predefined tools available to them.This fixed operational vocabulary motivates programmatic approaches that let models construct unique visual operations.
  • Conclusion and Future Frontiers: Future work targets better tools, broader coverage, standardized interfaces, autonomous tool composition, and internalized visual skills.The roadmap extends from selecting tools toward composing novel sequences and absorbing common tool functionality as intrinsic capability.

4 Stage 2: Programmatic Visual Manipulation

Stage 2 advances beyond selecting from a fixed toolkit by teaching models to create executable programs for tailored visual operations. Its methods are organized by composable operations and application scenarios.

  • Programmatic Visual Manipulation: Stage 2 turns the model from a tool orchestrator into a visual programmer that generates executable code for novel, task-specific visual operations.These programs enable customized analysis pipelines tailored to unique demands.
  • Programmatic Visual Manipulation: Table 2 categorizes programmatic methods as prompt-based or training-based, with † marking reinforcement-learning methods and the remaining methods using supervised fine-tuning.The table also organizes methods by visual, logical, and composite operations across perception, reasoning, and generation scenarios.

4.1 The Programmatic Leap: From Selection to Creation

Programmatic visual reasoning replaces fixed tool selection with generated code, giving models compositional flexibility, dynamic control, and interpretable reasoning traces.

  • Generated code expands visual reasoning beyond a finite toolbox into a generative grammar of composable operations.This supports novel, task-specific pipelines assembled from primitive functions.
  • Code enables complex visual workflows by composing primitives for multi-step criteria that predefined tools cannot address.An example matches object colors against the average color of a specified region.
  • Programmatic reasoning supports if-else branches, loops, and stateful variables that adapt execution to intermediate visual findings.This creates dynamic rather than strictly linear tool sequences.
  • Generated code provides a deterministic, human-readable reasoning trace with explicit and verifiable problem-solving steps.This improves transparency and control relative to opaque internal reasoning or black-box tool calls.

4.2 Formulation

The formulation represents programmatic visual reasoning as generating code segments, composing them into a program, executing that program on an image, and using the resulting image or outputs for continued reasoning.

  • The model generates code segments C_t conditioned on reasoning history, the initial image, and the input query.These segments form the programmatic actions at each reasoning step.
  • The segments are composed into a complete program C = (C_1, C_2, . . . , C_N) and passed to an external execution engine.The interpreter applies the program’s operations to the initial image I_0 = I.
  • Execution produces a potentially modified image I_final and textual or structured output O_prog, which the model uses to continue reasoning or answer.Outputs can include calculated values or detected features.
  • Unlike tool selection, code generation lets the model compose and define tailored visual operations through executable programs.This supports complex algorithms designed for the task at hand.

4.3 Categories of Composable Operations

Composable programmatic operations form a vocabulary for visual reasoning, including functions that inspect, transform, and alter image data.

  • The programmatic vocabulary consists of composable building blocks generated as code.These operations are organized into three main categories in Figure 3.
  • Visual operations directly interact with image data through detection, feature extraction, geometric transformation, and drawing functions.Examples include find_objects, crop_region, get_color, OCR, rotate, and resize.

4.4 Implementation Approaches

Implementation approaches span prompting, supervised fine-tuning, reinforcement learning, visual scratchpads, and increasingly integrated systems that generate, execute, and learn from visual programs.

  • Prompt-based approaches: Prompt-based methods generate executable Python code as transparent intermediate reasoning steps for deterministic, verifiable visual pipelines.VisProg pioneered decomposing natural-language tasks into modular operations invoking computer-vision tools.
  • Visual sketching: Visual sketching methods generate intermediate visual aids that inform later reasoning through iterative thought, action, and observation.Visual Sketchpad uses code-generated aids such as auxiliary geometry lines.
  • Integrated systems: Advanced systems combine orchestrator agents, specialized multimodal models, and vision experts to gather detailed evidence across components.VipAct exemplifies this multi-agent integration.
  • SFT-based approaches: SFT teaches models either to generate code as intermediate reasoning or to understand images rendered from code, encouraging structured problem-solving.Code-derived training data bridges symbolic program logic and sub-symbolic visual patterns.
  • Reinforcement learning: Reinforcement learning optimizes code-based operational sequences using execution feedback and task-oriented rewards rather than static imitation alone.Visual-ARFT addresses sparse valid-program spaces with verifiable rewards, while related systems interleave web search with image-processing code.

4.5 Conclusion and Future Frontiers

Programmatic visual manipulation offers flexible, interpretable reasoning by generating tailored code, but remains dependent on external execution and motivates intrinsic operations.

  • Conclusion: Generated programs combine primitive operations into tailored algorithms for complex visual problems while providing a transparent reasoning trace.The trace supports debugging and human–AI collaboration.
  • Conclusion: External code interpreters or search APIs create efficiency bottlenecks and fragile failure points.
  • Conclusion: The remaining semantic gap between latent reasoning and rigid code syntax motivates visual operations performed intrinsically.
  • Future Frontiers: Future work targets code debugging, visual diagnosis of incorrect outputs, and autonomous rewriting for more resilient agents.

5 Stage 3: Intrinsic Visual Imagination

Intrinsic visual imagination integrates visual generation into reasoning, enabling models to create and reuse visual states internally rather than outsourcing operations. The stage includes latent, explicit, and multimodal approaches, with future work targeting efficient, grounded, and collaborative imagination.

  • 5 Stage 3: Intrinsic Visual Imagination: Stage 3 closes the reasoning–perception loop by generating visual states internally as part of the model’s own thought process.
  • 5.2 Key Innovations: Intrinsic imagination unifies generative and reasoning capabilities, reducing the separation, latency, and information loss associated with external APIs or interpreters.
  • 5 Stage 3: Intrinsic Visual Imagination: A generated image becomes the next perceptual input, enabling a seamless loop between perception, reasoning, and imagination.
  • 5.3 Primary Paradigms: Implicit latent reasoning manipulates abstract features or visual tokens, whereas explicit visual reasoning generates human-readable intermediate images as a visual scratchpad.
  • 5.4 Training Methods: Supervised fine-tuning teaches interleaved multimodal thought, visual edits, and intermediate representations, while reinforcement learning also supports internal simulation.
  • 5.4.1 SFT-Based Approaches: Visual scratchpads can make multimodal reasoning more interpretable and robust by externalizing visual and spatial reasoning.
  • 5.5 Conclusion and Future Frontiers: Future research should explore abstract visual representations and dynamic world models for computationally efficient, generalizable, physically plausible reasoning.

6 Evaluations & Frameworks for Thinking with Images

Evaluation and implementation frameworks turn “Thinking with Images” into an empirical research domain spanning diverse reasoning tasks and prompt-, SFT-, and RL-based methods. Existing benchmarks report advantages for image-based reasoning, while future benchmarks should better test visual manipulation and imagination.

  • 6.1 Evaluations Benchmarks: Benchmarks span mathematical, STEM, perception, code, chart and table, real-world, and puzzle or game reasoning domains.
  • 6.1 Evaluations Benchmarks: Code-generation benchmarks test whether models can reconstruct visual artifacts’ structural logic as precise, executable programs.
  • 6.2 Implementation Frameworks: Prompting, supervised fine-tuning, and reinforcement learning provide implementation frameworks for enabling image-based reasoning with minor adjustments.
  • 6.3 Conclusion and Future Frontiers: These evaluations increasingly move from single-turn tasks toward multi-step, multi-task reasoning.
  • 6.3 Conclusion and Future Frontiers: Few current benchmarks require the multi-step manipulation and imagination needed to reach final answers.

7 Applications

Applications of “Thinking with Images” extend across interfaces, robotics, science, medicine, and education by using visual states as intermediate reasoning resources. These systems support closed-loop action, visual planning, domain-specific analysis, and interactive visual instruction, while safety and domain grounding remain important boundaries.

  • 7.1 Interactive Systems and User Interfaces: GUI agents use direct pixel perception and closed-loop reasoning to interpret affordances and select actions across web, desktop, and mobile environments.
  • 7.2 Embodied AI and Robotics: Visual subgoals and generated-image sequences improve planning in manipulation and navigation tasks, with image-only reasoning outperforming language-based methods in some spatial problems.
  • 7.3 AI for Science: Scientific applications show promise for automating data analysis and hypothesis generation, but require curated domain data to learn scientific priors general models miss.
  • 7.4 Healthcare and Medicine: Medical visual reasoning emphasizes verifiability and safety through expert knowledge and structured workflows.
  • 7.5 Education and Training: Educational agents can generate diagrams, highlight features, and sketch solutions in real time, enabling personalized visual instruction.

8 Future Directions

Future work must make visual reasoning more efficient, cognitively adaptive, secure, trustworthy, and evaluable while integrating visual thinking into unified agent architectures.

  • Efficiency: Current visual reasoning methods are computationally expensive because sequential tool calls and intermediate images add delay, limiting real-time use.Future research should preserve powerful visual reasoning without costly execution.
  • Efficiency: Cognitive efficiency requires models to choose between quick textual responses and deeper visual analysis according to task demands.Systems should avoid applying expensive reasoning to simple problems.
  • Efficiency: Built-in latent-space modules for common visual tasks could reduce the cost of external tools and full-pixel processing.Candidate modules include object search and element comparison.
  • Safety and Trust: Visual reasoning introduces misuse risks because sequentially generated images can support false claims and automated disinformation campaigns.Safety must address manipulation of the entire visual thinking process, not only harmful individual images.
  • Safety and Trust: Security threats extend across the reasoning loop, including poisoned tools and inputs that redirect attention or tool selection.The expanded attack surface includes both external components and intermediate decisions.
  • Safety and Trust: Black-box visual reasoning can hide biases in intermediate steps, requiring greater transparency and monitoring for trustworthy systems.The cited example involves stereotypical or selectively critical image generation for certain groups.
  • Evaluation: Existing benchmarks often assess only final answers, so future evaluations must test visual grounding, logical coherence, and process necessity.Answer accuracy alone can allow correct answers produced through shortcuts.
  • Unified Architectures: A unified visual thinker should use metareasoning to select among immediate inference and three stages of visual cognition within a feedback-enabled workspace.The architecture connects internal visual thought to textual, visual, or executable outputs and positions visual thinking as a cognitive engine for agents.

9 Conclusion

The survey frames Thinking with Images as a three-stage progression from external tools to programmatic manipulation and intrinsic imagination, transforming vision into a dynamic cognitive workspace. It concludes that the field remains early, with efficiency, robustness, and generalization still requiring substantial advances.

  • Conclusion: The survey organizes Thinking with Images into three stages: external-tool use, programmatic manipulation, and intrinsic imagination.This progression changes vision from a static input into a dynamic cognitive workspace.
  • Conclusion: The field remains in its early stages, with efficiency, robustness, and generalization still presenting significant challenges.Future directions include richer world models, abstract visual representations beyond pixels, and unified human-AI collaboration frameworks.

Contributions

The authors divide contributions across the survey’s conceptual sections, core methods, visual elements, tables, supervision, and manuscript refinement.

  • Authorship: Zhaochen Su drafted the abstract, introduction, foundations, and introductory sections for each stage.
  • Authorship: Core-method sections were divided by approach, with prompting, supervised fine-tuning, and reinforcement-learning sections assigned to different authors.
  • Visual Contributions: Zhaochen Su created the abstract figure and Figures 2, 3, and 5, while other authors contributed Figures 1 and 4 and visual assets.
  • Visual Contributions: Tables 1, 2, and 3 were produced collaboratively by Zhaochen, Zhenhua, Peng, Yan, and Kaide.
  • Supervision and Feedback: Junxian He and Yi R. (May) Fung supervised the project, while four additional authors provided feedback and critical suggestions.
Loading 2506.23918v3…