Source-linked AI summary
Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, Linjie Li, Yu Cheng, Heng Ji, Junxian He, Yi R. Fung
TL;DR
Text-centric multimodal reasoning treats vision as static context, leaving a gap between rich visual information and symbolic thought. This survey organizes the emerging “Thinking with Images” paradigm into three stages and concludes that dynamic visual reasoning advances multimodal AI toward deeper analysis and physical simulation.
Problem
Text-only reasoning after a static visual encoding leaves a semantic gap between rich, continuous visual information and discrete symbolic language.
Source: §1 · p. 4, §2.1 · p. 6Method
The survey structures “Thinking with Images” into three stages: external tool use, programmatic manipulation, and intrinsic imagination.
Source: §1 · p. 4, §9 · p. 36Results
The survey finds that visual scratchpads, unified models, and interleaved multimodal dialogue collectively advance transparent, grounded visual reasoning.
Source: §3.4 · p. 14, §5.5 · p. 24, §6.3 · p. 28Takeaways & Limitations
Thinking with Images reframes vision as a dynamic cognitive workspace supporting deeper perceptual analysis and more robust physical simulation.
Source: §5.5 · p. 24, §9 · p. 36Visual reasoning remains computationally expensive because intermediate images require dense patch processing, making long reasoning paths practically infeasible.
Source: §2.4 · p. 9
Abstract
from arXiv · showhide
Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-centric approach, however, treats vision as a static, initial context, creating a fundamental "semantic gap" between rich perceptual data and discrete symbolic thought. Human cognition often transcends language, utilizing vision as a dynamic mental sketchpad. A similar evolution is now unfolding in AI, marking a fundamental paradigm shift from models that merely think about images to those that can truly think with images. This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process, transforming vision from a passive input into a dynamic, manipulable cognitive workspace. In this survey, we chart this evolution of intelligence along a trajectory of increasing cognitive autonomy, which unfolds across three key stages: from external tool exploration, through programmatic manipulation, to intrinsic imagination. To structure this rapidly evolving field, our survey makes four key contributions. (1) We establish the foundational principles of the think with image paradigm and its three-stage framework. (2) We provide a comprehensive review of the core methods that characterize each stage of this roadmap. (3) We analyze the critical landscape of evaluation benchmarks and transformative applications. (4) We identify significant challenges and outline promising future directions. By providing this structured overview, we aim to offer a clear roadmap for future research towards more powerful and human-aligned multimodal AI.
1 Introduction
The survey frames “Thinking with Images” as a shift from static, text-mediated visual reasoning to dynamic visual representations that models can query, manipulate, generate, and verify as intermediate thoughts. It organizes this emerging paradigm by increasing cognitive autonomy, from external tool use through programmatic manipulation to intrinsic imagination, while reviewing its methods, evaluations, applications, challenges, and future directions.
1 Introduction
- Text-centric “Thinking about Images” treats vision as fixed context and language as the exclusive reasoning medium, creating an information bottleneck for iterative visual tasks.This limitation affects complex physical reasoning, precise spatial manipulation, and long-horizon planning.Source: §1 · p. 4
- “Thinking with Images” makes visual representations manipulable and verifiable intermediate steps, turning vision into a dynamic cognitive workspace rather than a passive input.Models can actively see, query, modify, and generate visual information during reasoning.Source: §1 · p. 4
- The survey’s taxonomy traces increasing cognitive autonomy from commanders orchestrating external tools, through visual programmers generating tailored operations, to visual thinkers capable of intrinsic imagination and simulation.Figure 1 organizes these stages alongside supporting benchmarks, frameworks, and applications.Source: §1 · p. 4, Figure 1 · p. 5
- The survey systematically covers the paradigm’s foundations, stage-specific methodologies, evaluation benchmarks, implementation frameworks, applications, challenges, and future directions.Its stated aim is to provide a clear taxonomy and inspire more powerful, intuitive, and multimodal AI.Source: §1 · p. 4
1.1 The Position of Our Survey
- Unlike prior reviews that largely retain passive, text-centric visual reasoning, this survey is positioned as the first systematic account of the shift toward dynamic visual cognition and closed-loop intrinsic imagination.It characterizes the progression from tool orchestration to programmatic visual analysis and intrinsic visual information within a cognitive loop.Source: §1.1 · p. 5
2 Foundations of the Thinking with Images Paradigm
“Thinking with Images” reframes vision from a static input into a dynamic, manipulable workspace that supports exploration, structured reasoning, and generative planning. The paradigm progresses from external tools through programmatic manipulation to intrinsic imagination, while introducing formal and practical challenges in cost, robustness, architecture, and generalization.
2.1 The Cognitive Imperative: Why We Need “Thinking with Images”?
- Thinking with Images closes the semantic gap of text-only reasoning by enabling dynamic exploration, visual scratchpad reasoning, and goal-oriented simulation.Models can iteratively zoom or highlight regions, render auxiliary constructions, and simulate future actions whose visual inconsistencies provide error signals.Source: §2.1 · pp. 6–7
2.2 How Does It Work? The Three-Stage Evolution
- The three-stage framework spans tool-driven exploration, programmatic visual manipulation, and intrinsic visual imagination, reflecting increasing autonomy but not a strictly linear progression.Stage 1 selects predefined tools; Stage 2 generates executable visual-analysis code; Stage 3 generates and critiques intermediate images within a closed loop, while efficient systems may select different stages per task.Source: §2.2 · pp. 7–8
2.3 What is Thinking with Images? A Formal Definition
- Formally, Thinking with Images samples each reasoning step from textual or visual spaces, allowing the evolving history to contain both language and intermediate visual artifacts.Unlike Thinking about Images, where visual features are encoded once as fixed context, intermediate artifacts may come from tools, code-generated visualizations [Gupta and Kembhavi, 2023], or intrinsic predictions [Xu et al., 2025].Source: §2.3 · p. 8
2.4 Unique Challenges
- Visual thought faces prohibitive computational cost and severe error propagation because dense patch processing compounds across intermediate images and flawed perception can establish a false ground truth.These constraints limit the depth of visual deliberation and can redirect subsequent reasoning toward a coherent but irrelevant interpretation.Source: §2.4 · p. 9
- Modular vision-language architectures can impede iterative visual thinking by translating spatial information into sequential form, motivating more direct interaction between reasoning engines and visual canvases.The separation risks losing fine-grained detail and makes perception of the model’s actions indirect [Liu et al., 2024a, Bai et al., 2025b, Team, 2024].Source: §2.4 · p. 9
- Cross-task generalization remains difficult because geometry, diagnosis, and navigation require constructive, analytical, and simulative strategies, respectively.Robust agents therefore need diverse visual strategies and a meta-policy for selecting among them [Wang et al., 2025d, Zheng et al., 2025, Li et al., 2025a].Source: §2.4 · p. 9
3 Stage 1: Tool-Driven Visual Exploration
Stage 1 reframes visual reasoning as active orchestration of predefined external tools, progressing from language-mediated prompting through supervised procedural learning to reinforcement-learned policies. Its transparency and practicality are balanced by fixed-tool limitations that motivate programmatic visual manipulation and more autonomous skill internalization.
3 Stage 1: Tool-Driven Visual Exploration
- Tool-Driven Visual Exploration makes the model an active planner that selects and sequences predefined visual tools, replacing static visual context with controlled visual inquiry.Tool outputs—including modified images, segmentation masks, or structured text—return to the model’s state to inform subsequent reasoning.Source: §3 · p. 9, §3.1 · p. 10
3.1 Formulation
- Stage 1 operates by generating symbolic tool calls from a fixed inventory, so intelligence lies in logical orchestration rather than inventing new visual operations.This finite operational vocabulary marks the central distinction from “Thinking about Images,” where visual input remains static.Source: §3.1 · pp. 9–10
3.2 Categories of Visual Tools
- The toolkit spans semantic grounding, perception exploration, and visual reasoning aids that respectively translate pixels, inspect details, and structure thought on the image.Examples include captioning and OCR, cropping, zooming, detection and segmentation, plus highlighting, masking, auxiliary lines, and counting pointers [Hu et al., 2022] [Ma et al., 2024] [Shen et al., 2024, Zheng et al., 2025] [Wu et al., 2024c, Yang et al., 2023b] [Fu et al., 2025a].Source: §3.2 · pp. 10–11, Figure 3 · p. 10
3.3 Implementation Approaches
- Implementation combines prompt-based coordination, supervised fine-tuning for procedural competence, and reinforcement learning for policy optimization through interaction and reward.These methods respectively support in-context orchestration, demonstrated tool-use trajectories, and discovery of effective visual-reasoning strategies.Source: §3.3 · p. 11, §3.3.1 · p. 11, §3.3.2 · p. 13
3.3.1 Prompt-Based Approaches
- Prompt-based approaches coordinate visual capabilities without parameter updates through language mediation, input manipulation, structured exploration, and specialized visual experts.Examples include Socratic Models [Zeng et al., 2022], PromptCap [Hu et al., 2022], MM-REACT [Yang et al., 2023a], ZoomEye [Shen et al., 2024], and Visual Thoughts [Cheng et al., 2025].Source: §3.3.1 · pp. 11–13, Table 1 · p. 12
3.3.2 SFT-Based Approaches
- Supervised fine-tuning teaches models to invoke and integrate external tools, develop internal visual operations, and actively allocate attention during reasoning.It supports tool-use trajectories, intrinsic grounding and cropping, region replay, masking, cross-image comparison, and selective visual revisitation [Liu et al., 2023] [Ma et al., 2024] [Qi et al., 2024] [Zheng et al., 2024a].Source: §3.3.2 · p. 13
3.3.3 RL-Based Approaches
- Reinforcement learning progresses from generalizable visual skills to multimodal grounded rationales and active orchestration of tools, including adaptive zooming, region selection, and drawing-based reasoning.Representative systems include Jigsaw-R1 [Wang et al., 2025c], GRIT [Fan et al., 2025a], VisTA [Huang et al., 2025a], DeepEyes [Zheng et al., 2025], and OpenThinkIMG [Su et al., 2025a].Source: §3.3.3 · p. 14
3.4 Conclusion and Future Frontiers
- Tool-driven exploration is transparent and practical, but fixed predefined tools prevent novel operations; future work targets better tools, autonomous composition, adaptation, and skill internalization.The paradigm’s methodological spectrum runs from zero-shot prompting to procedural SFT and autonomous RL policies [Yang et al., 2023a, Ma et al., 2024].Source: §3.4 · pp. 14–15
4 Stage 2: Programmatic Visual Manipulation
Stage 2 marks a shift from selecting fixed tools to generating executable code for tailored visual operations, enabling novel analysis pipelines for unique demands.
4 Stage 2: Programmatic Visual Manipulation
- Stage 2 transforms models from fixed-tool orchestrators into visual programmers that generate executable code for tailored visual operations and novel analysis pipelines.Table 2 categorizes the stage’s main methods by composable operations and primary application scenarios.Source: §4 · pp. 15–16
4.1 The Programmatic Leap: From Selection to Creation
Programmatic visual reasoning replaces a finite tool menu with code generation, enabling compositional visual operations, adaptive execution, and interpretable human-controllable reasoning traces.
Compositional Flexibility
- Code generation shifts visual reasoning from a finite toolbox to a generative grammar, enabling novel multi-step pipelines and first-principles solutions for complex criteria [Gupta and Kembhavi, 2023, Surís et al., 2023].This supports potentially infinite visual operations, such as finding objects whose color matches a region’s average color.Source: §4.1 · p. 16
Dynamic and Non-Linear Control Flow
- Generated programs embed branches, loops, and stateful variables, allowing execution to adapt to intermediate visual findings [Chen et al., 2022b, Gao et al., 2023, Yao et al., 2023, Hu et al., 2024b].For example, the model can diagnose blur, generate deblurring code, and then continue analysis [Liu et al., 2025d].Source: §4.1 · p. 16
Enhanced Interpretability and Control
- Generated code provides a deterministic, human-readable reasoning trace whose explicit steps can be inspected, debugged, refined, and used for collaborative control [Gupta and Kembhavi, 2023, Hu et al., 2024a, Mallis et al., 2024, Chen et al., 2025d].This transparency supports iterative human guidance and correction, especially in high-stakes visual applications.Source: §4.1 · p. 16
4.2 Formulation
Section 4.2 formulates programmatic visual reasoning as iterative code generation conditioned on the reasoning state, image, and query. Generated programs are executed externally to produce modified images and structured outputs that guide continued reasoning or answer generation.
Formulation
- The LMM generates code segments C_t conditioned on prior code and outputs, the initial image I, and query Q, composing them into a complete program C.The state S_t records previously generated code snippets and outputs; implementations commonly use Python with computer-vision libraries or domain-specific APIs.Source: §4.2 · p. 16
- An external interpreter E_exec applies the program’s operations to the initial image I_0 = I, yielding a modified image I_final and textual or structured output O_prog.These outputs can include calculated values or detected features and are fed back into continued reasoning or answer generation.Source: §4.2 · p. 16
- Unlike Stage 1, the LMM defines and composes visual operations through code generation, enabling tailored and complex algorithms.The distinction is that visual operations are programmatically selected and composed by the model rather than merely selected.Source: §4.2 · p. 16
4.3 Categories of Composable Operations
Programmatic visual reasoning uses code-generated, composable building blocks organized into visual, logical, and composite operations. Together, these operations let models manipulate images, structure stateful reasoning, and construct tailored algorithms for complex visual tasks.
4.3 Categories of Composable Operations
- Programmatic reasoning organizes code-generated building blocks into three categories: visual, logical, and composite operations.These categories define the model’s operational vocabulary for visual computation.Source: §4.3 · p. 17
Visual Operations
- Visual operations perceive and manipulate images through detection, feature extraction, geometric transformations, and drawings that guide reasoning [Bradski, 2000, Chen et al., 2025d].Examples include find_objects, crop_region, get_color, ocr, rotate, resize, and image annotation.Source: §4.3 · p. 17
Logical Operations
- Logical operations provide variables, conditionals, and loops, enabling stateful, multi-step visual reasoning [Hu et al., 2024a, Ge et al., 2025].They act as the grammar that organizes visual operations into computational processes.Source: §4.3 · p. 17
Composite Operations
- Composite operations combine visual and logical primitives to construct problem-specific algorithms, such as finding red circles larger than the average circle area [Fan et al., 2024, Mallis et al., 2024].The example requires iteration, property checks, stateful calculations, and result selection, demonstrating computation beyond predefined capabilities.Source: §4.3 · p. 17
4.4 Implementation Approaches
Programmatic visual reasoning adapts training-free prompting, supervised fine-tuning, and reinforcement learning to equip multimodal models with code-generation capabilities. These methods progress from interpretable code-based visual analysis and manipulation to learned programmatic logic and autonomous feedback-driven workflows.
4.4 Implementation Approaches
- Implementation approaches span training-free prompting, SFT, and RL, each adapting programmatic methods to equip LMMs with code-generation capabilities.Source: §4.4 · p. 17
4.4.1 Prompt-Based Approaches
- Prompt-based methods generate executable Python as transparent intermediate reasoning, supporting modular visual computation, iterative visual sketching, and specialized multi-agent systems.VisProg [Gupta and Kembhavi, 2023] and ViperGPT [Surís et al., 2023] establish training-free, interpretable program composition, while Visual Sketchpad [Hu et al., 2024b], ReFocus [Fu et al., 2025a], SketchAgent [Vinker et al., 2024], Interactive Sketchpad [Chen et al., 2025d], VipAct [Zhang et al., 2024a], MMFactory [Fan et al., 2024], and CAD-Assistant [Mallis et al., 2024] extend manipulation and system integration.Source: §4.4.1 · pp. 17–18
4.4.2 SFT-Based Approaches
- SFT distills programmatic logic or code-rendered images into LMMs, inducing more structured and compositional visual reasoning through code-derived training data.VPD [Hu et al., 2024a] converts verified program traces into rationales for single-pass reasoning, while MathCoder-VL [Wang et al., 2025d], PROVISION [Zhang et al., 2024b], CoSyn [Yang et al., 2025], and VLM [Ge et al., 2025] use image-code or code-synthesized data for alignment and instruction tuning.Source: §4.4.2 · pp. 18–19
4.4.3 RL-Based Approaches
- RL replaces static imitation with feedback-driven discovery, teaching models to generate functional visual operations and interleave web search with code for open-world reasoning.Visual-ARFT [Liu et al., 2025d] uses intermediate execution-success rewards to address sparse valid-program spaces, shifting models from tool users to tool makers.Source: §4.4.3 · pp. 19–20
4.5 Conclusion and Future Frontiers
Programmatic visual manipulation offers compositional flexibility and interpretable reasoning traces but remains constrained by external execution and a persistent semantic gap. Future research should improve robustness, unify tool use with code generation, and ultimately enable intrinsic visual imagination.
Conclusion
- Programmatic visual manipulation combines compositional flexibility with interpretable, verifiable reasoning traces, supporting complex visual problems beyond fixed toolsets.Generated code constructs algorithms from primitive operations and provides a transparent trace for debugging and human-AI collaboration.Source: §4.5 · p. 20
- Its reliance on external interpreters or search APIs creates efficiency bottlenecks and fragile failure points, while code syntax remains semantically distant from latent reasoning.These limitations motivate intrinsic visual imagination, in which models generate new visual states rather than instructions.Source: §4.5 · p. 20
Future Frontiers
- Future programmatic systems should self-correct generated code by interpreting execution errors or visually diagnosing incorrect outputs before rewriting programs.Such capabilities are intended to create more resilient and reliable agents.Source: §4.5 · p. 20
- Unified agents should combine Stage 1 tool orchestration with Stage 2 code generation, strategically invoking high-level tools as program subroutines.This approach addresses the trade-off between specialized tools’ pragmatic efficiency and programmatic creation’s unbounded flexibility.Source: §4.5 · p. 20
5 Stage 3: Intrinsic Visual Imagination
Stage 3 makes visual generation an intrinsic part of reasoning, closing the loop between perception, language, and internally simulated imagery. Its methods span latent, explicit, and interleaved visual thought, trained primarily through supervised fine-tuning and reinforcement learning, while future work targets abstract representations, world models, and human-AI co-imagination.
5 Stage 3: Intrinsic Visual Imagination
- Intrinsic visual imagination gives models a closed cognitive loop in which generated visual states become native intermediate thoughts rather than externally executed instructions.This stage represents a shift from visual programming to internally simulated visual cognition.Source: §5 · pp. 20–21
5.1 The Architectural Leap: From Execution to Imagination
- Unified generation and reasoning remove external-call bottlenecks, while visual simulation replaces explicit coordinate-based procedures with holistic images that can implicitly enforce physical constraints [Team, 2024, Sun et al., 2024a, Xu et al., 2025].The architectural shift internalizes visual content creation and supports more fluid reasoning about complex physical and spatial problems.Source: §5.1 · p. 21
5.2 Formulation
- At each reasoning step, the model may generate an image directly, feed it back as the next perceptual state, and continue a seamless perception–reasoning–imagination loop.The generated image is produced by internal generative mechanisms rather than an outsourced function.Source: §5.2 · p. 21
5.3 Categories of Generative Thoughts
- The three generative-thought paradigms are implicit latent reasoning, explicit visual reasoning through human-readable scratchpads, and interleaved text-image dialogue [Team, 2024, Sun et al., 2024a, Guo et al., 2025a, Li et al., 2025a, Xu et al., 2025, Duan et al., 2025, Chern et al., 2025, Zhao et al., 2025a].They trade off efficiency and abstraction, transparency and interpretability, and synergistic language–vision reasoning, respectively.Source: §5.3 · pp. 21–22
5.4 Implementation Approaches
- SFT and RL repurpose the model’s generative architecture for internal simulation, with SFT teaching demonstrated multimodal reasoning patterns and RL discovering effective multi-step strategies.Together they shift learning from directing external tasks toward performing generative reasoning internally.Source: §5.4 · p. 22, §5.4.1 · p. 22, §5.4.2 · p. 23
5.4.1 SFT-Based Approaches
- SFT unifies understanding and generation through autoregressive mixed-modal training, visual scratchpads, and generation–reasoning integration across models such as Chameleon, Emu2, BLIP-3o, GILL, and BAGEL [Team, 2024, Sun et al., 2024a, Chen et al., 2025b, Koh et al., 2023, Deng et al., 2025b].It also trains explicit plans, visualizations, subgoals, and self-critiques, making multimodal reasoning more interpretable and robust [Fang et al., 2025, Li et al., 2025a, Zhao et al., 2025a, Chern et al., 2025].Source: §5.4.1 · pp. 22–23
5.4.2 RL-Based Approaches
- RL enables pure visual planning, text-orchestrated imagination, and reflective reward modeling, including generated-image planning [Xu et al., 2025], bi-level CoT [Duan et al., 2025, Jiang et al., 2025a], and PARM/PARM++ self-correction [Guo et al., 2025a].These methods support future-state simulation, semantic-to-pixel planning, semantic control, and stepwise evaluation of imagined thoughts.Source: §5.4.2 · p. 23
5.5 Conclusion and Future Frontiers
- Stage 3 advances cognitive autonomy through scalable unified models, interpretable visual scratchpads, and interleaved multimodal dialogue, while future work targets abstract visual representations, dynamic world models, and human-AI co-imagination.These frontiers aim to improve efficiency, generalization, physical plausibility, collaboration, and safeguards for realistic visual fabrication [Weng et al., 2025a, Zhou et al., 2023, 2024b, Jiang et al., 2025b, Hao et al., 2023, Chen et al., 2025d].Source: §5.5 · p. 24
6 Evaluations & Frameworks for Thinking with Images
This section surveys benchmarks and implementation frameworks that turn “Thinking with Images” into an assessable and reproducible research domain. It spans seven evaluation categories, prompt-, SFT-, and RL-based frameworks, and future directions for visual manipulation and scalable learning.
6.1 Evaluations Benchmarks
- Benchmarks span seven domains—mathematical reasoning, STEM, perception reasoning, code generation, chart and table reasoning, real-world applications, and puzzles and games—covering complex visual reasoning beyond static perception.Figure 4 illustrates these seven evaluation domains and their applications.Source: §6.1 · p. 24, Figure 4 · p. 25
- Mathematical and STEM benchmarks test multi-step reasoning with constructed visual content, physical and causal processes, and increasingly advanced educational, university-level, and professional knowledge.Mathematical evaluations include geometry, algebra, analytic geometry, arithmetic, robustness to diagram variations, minimal textual context, and structured outputs; STEM tasks assess dynamic phenomena and internal world models.Source: §6.1 · pp. 24–25
- Other benchmarks assess strategic planning, fine-grained perception, visual-to-code reconstruction, structured-data analysis, and commonsense reasoning over noisy real-world images.These tasks include long-horizon game simulation, hallucination detection, executable webpage generation, chart transformations, and practical temporal or numerical reasoning.Source: §6.1 · p. 26
6.2 Implementation Frameworks
- Prompt-based frameworks provide lightweight tool integration and agent coordination, while specialized systems such as MM-REACT, VisProg [Gupta and Kembhavi, 2023], ViperGPT [Surís et al., 2023], Visual Sketchpad [Hu et al., 2024b], and ReFocus [Fu et al., 2025a] support interactive visual reasoning.These systems orchestrate vision experts, use generated code as reasoning plans, or generate and modify images during interaction.Source: §6.2 · p. 27
- SFT frameworks scale and customize training, while specialized structured data teaches tool invocation, program execution traces, and generative visual thought beyond instruction following.Unsloth achieves up to 60% memory reduction and 30x speedup, while other frameworks target fine-grained control, distributed training, or enterprise hardware optimization.Source: §6.2 · p. 27
- RL frameworks provide scalable training infrastructure, and specialized methods learn tool orchestration, direct image manipulation, or routing among perception heads for visual reasoning.OpenThinkIMG [Su et al., 2025a], VILASR [Wu et al., 2025a], and VisionReasoner [Liu et al., 2025b] extend general-purpose RL with these capabilities.Source: §6.2 · pp. 27–28
6.3 Conclusion and Future Frontiers
- Future work should benchmark visual manipulation and construction while improving prompting, SFT, and RL efficiency, scalability, and supervision from intermediate reasoning steps.Proposed tasks include drawing auxiliary geometry lines, sketching maze paths, generating intermediate visual states, embedding lightweight visual-tool commands, and using code-execution feedback beyond sparse outcome rewards.Source: §6.3 · p. 28
7 Applications
“Thinking with Images” expands multimodal applications by making visual information an active workspace for interaction, physical action, scientific analysis, clinical reasoning, and education. Across these domains, systems increasingly use visual perception, manipulation, intermediate reasoning, and imagination to support more capable and human-aligned behavior.
7 Applications
- LMMs apply “Thinking with Images” across interactive systems, embodied agents, science, healthcare, and education by turning visual information into an active basis for reasoning and action.The applications span intuitive human-machine interaction, visual analysis, physical control, clinical decision-making, and interactive learning.Source: §7 · p. 28
7.1 Interactive Systems and User Interfaces
- GUI agents now reason directly from pixels in closed loops, observing interface affordances, planning actions, and operating across web, desktop, and mobile environments without HTML representations.CogAgent achieves state-of-the-art GUI benchmark results and can outperform agents using extracted HTML; ShowUI, UI-TARS, and Aguvis exemplify screenshot-based visual-to-action control. Data generation through web exploration, tutorial conversion, and few-shot demonstrations addresses interface diversity.Source: §7.1 · p. 29
7.2 Embodied AI and Robotics
- Embodied agents use visual chains of thought and generated-image world models to plan physical actions, improve manipulation and generalization, and simulate future states for long-horizon decision-making.CoT-VLA predicts intermediate visual subgoals, Embodied ECoT reasons over plans, object features, and robot state, and VPRL plans through image sequences, outperforming language-based methods in spatial navigation; world models such as Genie, 3D-VLA, and UniSim support visual imagination.Source: §7.2 · pp. 29–30
7.3 AI for Science
- In science, “Thinking with Images” enables active analysis and visual simulation, but reliable performance requires domain-specific adaptation because general-world knowledge misses specialized scientific principles.Systems can quantify deforestation, segment cells, and simulate physical outcomes such as block-tower stability; curated physics and satellite-image datasets provide scientific priors for more advanced visual reasoning.Source: §7.3 · pp. 30–31
7.4 Healthcare and Medicine
- Healthcare systems are shifting from passive image interpretation toward collaborative, explainable clinical reasoning, with expert-guided visual workflows and structured processes prioritized for reliability, safety, and auditability.Anatomical ontologies guide multi-step scan searches, digital pathology systems construct visual cell-tissue hierarchies, MMedAgent-RL simulates specialist consultation, and HealthGPT generates comparison images within reasoning.Source: §7.4 · p. 31
7.5 Education and Training
- Educational agents combine language with generated diagrams, real-time sketches, and visual feedback to provide personalized interactive tutoring and context-aware professional training.Interactive Sketchpad illustrates STEM reasoning through code-generated diagrams, while VidAAS analyzes classroom recordings, including body language and spatial arrangements, to assess teaching performance.Source: §7.5 · pp. 31–32
8 Future Directions
Future work aims to make visual cognition more efficient, safe, evaluable, multimodal, and capable of flexible metareasoning. The proposed unified visual thinker combines adaptive control with a multi-stage visual workspace to connect internal thought and external action.
8.1 Computational and Cognitive Efficiency
- Efficient visual reasoning requires compressing multi-step processes, avoiding unnecessary intermediate images, and dynamically choosing between quick textual answers and deliberate visual analysis.Specialized latent-space modules could replace slower external tools and full-pixel rendering for common visual tasks.Source: §8.1 · p. 32
8.2 Safe and Trustworthy Visual Cognition
- Future systems must prevent visual reasoning from being hijacked into fabricated evidence, protect the entire reasoning loop, and expose internal biases for trustworthy cognition.Threats include automated disinformation narratives, poisoned tools, adversarial tool selection, and hidden stereotyping within visual thought.Source: §8.2 · pp. 32–33
8.3 Novel Benchmarks and Evaluation Methodologies
- Future benchmarks must require interactive, multi-step visual construction and evaluate whether reasoning steps are visually grounded, coherent, and necessary rather than merely accurate.Necessity can be tested by removing an intermediate visual step and checking whether the model still reaches the solution.Source: §8.3 · p. 33
8.4 Thinking with Audio, Video, and the World
- The paradigm should extend from static images to audio, video, and embodied interaction, requiring temporal navigation, internal simulation, and perception-action feedback.Video reasoning may reconstruct unseen events or predict future frames, while embodied agents rehearse actions before physical execution.Source: §8.4 · pp. 33–34
8.5.1 Thinking with Images v.s. Agentic Frameworks
- “Thinking with Images” is an internal deliberation mechanism, whereas agent frameworks organize external execution; together, visual cognition can serve as the agent’s reasoning engine.Their visual roles and intermediate artifacts differ, but the three autonomy stages can progressively empower intelligent action.Source: §8.5.1 · pp. 34–35
8.5.2 Blueprint for a Unified Visual Thinker
- A unified visual thinker uses metacognitive control to select immediate inference or a multi-stage visual workspace, enabling flexible and efficient multimodal reasoning.The workspace supports tool exploration, programmatic analysis, and intrinsic imagination, while translating thoughts into textual, visual, or executable outputs.Source: §8.5.2 · pp. 35–36, Figure 5 · p. 35
- The Metacognitive Controller assesses task complexity, routing simple problems directly and activating the Visual Cognitive Workspace when deeper visual reasoning is needed.This design avoids applying the most expensive reasoning process to every task.Source: §8.5.2 · p. 35
- The Visual Cognitive Workspace supports feedback-driven reasoning across three autonomy stages, while remaining able to stop early when a satisfactory solution is reached.This flexibility lets systems review generated thoughts without requiring a complete reasoning cycle.Source: §8.5.2 · pp. 35–36
9 Conclusion
The survey frames “Thinking with Images” as a three-stage evolution that transforms vision from static input into a dynamic cognitive workspace, while emphasizing that visual cognition still faces major challenges.
Conclusion
- The survey organizes Thinking with Images into three stages: external tool use, programmatic manipulation, and intrinsic imagination.This progression shifts vision from a static input toward a dynamic cognitive workspace that supports deeper perceptual analysis and more robust physical simulation.Source: §9 · p. 36
- Advancing visual cognition requires better efficiency, robustness, generalization, world models, abstract visual representations beyond pixels, and unified human-AI collaboration frameworks.The survey presents these directions as necessary steps toward AI agents that reason within the world using an integrated mind’s eye.Source: §9 · p. 36
Contributions
The paper’s sections and core methods were distributed among authors by topic and stage, while visual elements, tables, supervision, and manuscript feedback were contributed collaboratively.
- Authors divided drafting responsibilities by approach and stage, covering prompting, SFT, RL, stage framing, introductory sections, and later sections.Zhaochen Su drafted the abstract, introduction, foundations, and stage introductions; Zhenhua Liu, Peng Xia, and Yan Ma led prompting, SFT, and RL sections, respectively, while Hangyu Guo, Jiaqi Liu, and Zhaochen Su with Xiaoye Qu drafted later sections.Source: Contributions · p. 36
- Visual elements were created through a mix of individual and collaborative contributions, with all tables produced collaboratively by five authors.Zhaochen Su created the abstract figure and Figures 2, 3, and 5; Figure 1 involved Peng Xia, Zhenhua Liu, and Kaide Zeng; Figure 4 was contributed by Hangyu Guo; Yanshu Li provided visual assets and initial drafts.Source: Contributions · p. 36
- Junxian He and Yi R. (May) Fung supervised the project, while Zhengyuan Yang, Linjie Li, Yu Cheng, and Heng Ji provided manuscript feedback and critical suggestions.Source: Contributions · p. 36
Related papers
This paper cites
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale2020
- Gemini: A Family of Highly Capable Multimodal Models2023
- Constitutional AI: Harmlessness from AI Feedback2022
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control2023
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments2024
- PAL: Program-aided Language Models2022
Cited by
- OACap: Caption-Guided Active Observation for Detailed Video Captioning2026
- What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models2026
- Self-Conditioning Enables Deep Sequential Reasoning in Flow Matching2026
- From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models2026
- When Vision Meets Graphs: A Survey on Graph Reasoning and Learning2026
- Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents2026