PaperlayerView exact revision on arXiv

Source-linked AI summary

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

arXiv:2607.26769v1cs.CVcs.AI

TL;DR

Existing evaluations do not jointly establish whether models select relevant visual actions, faithfully render them, and depend on the resulting states. See2Think combines a benchmark and an intervenable process protocol, finding that visual reasoning effectiveness varies by model and environment rather than following one universally best strategy.

  • Problem

    Existing evaluations do not jointly determine whether self-generated visual actions are relevant, faithfully rendered, and behaviorally used in subsequent reasoning.

  • Method

    See2Think combines 1.2K visually dependent problems across 12 categories with VAoT, which records visual actions, rendered states, and subsequent reasoning.

  • Results

    No single inference setting dominates across tasks; the strongest setting varies with both the model and the visual-reasoning task.

  • Takeaways & Limitations

    Intermediate visual reasoning is a conditional capability rather than a universal accuracy booster, with faithful realization and visual-evidence use remaining bottlenecks.

  • Takeaways & Limitations

    The evaluation covers four representative models, relies on external rendering and automatic process evaluation, and measures intervention effects whose magnitude varies with filtering criteria.

Abstract

from arXiv · show

Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.

1 Introduction

See2Think evaluates whether multimodal models genuinely use intermediate visual states rather than merely producing them. It combines a visually dependent benchmark with an observable, intervenable protocol that diagnoses visual actions, rendering, and downstream feedback use.

  • Motivation: Existing evaluations often emphasize final answers or textual rationales, while task collections may lack visual diversity or permit text-only shortcuts.Visual-thinking benchmarks also commonly rely on end-task performance, oracle visual clues, or partial process measures.
  • See2ThinkBench: See2ThinkBench contains 1.2K visually dependent reasoning problems across 12 task categories and three visual environments.Caption-only solvability filtering reduces text-dominant shortcuts, and answer-format normalization supports consistent evaluation across free-form and structured outputs.
  • Diagnostic framing: See2Think distinguishes whether an intermediate visual state is useful from whether model behavior actually depends on it.This formulation treats genuine visual-state use as a controlled diagnostic problem.
  • Visual Action-of-Thought: VAoT records visual actions, rendered states, and their downstream use through an observable and intervenable protocol.The protocol supports diagnosis of action relevance, render faithfulness, and feedback uptake, including controlled tests of returned visual states.
  • Findings: Experiments across four representative proprietary and open-source models reveal no universally optimal visual-reasoning strategy and expose when rendered feedback helps, harms, or merely decorates reasoning.The framework also identifies a persistent bottleneck after action selection.

2 See2ThinkBench

See2ThinkBench is a 1.2K-sample benchmark spanning 12 visually dependent reasoning categories across 2D structured, 3D scene, and real-world visual reasoning. It filters text-solvable and ambiguous cases, converts tasks toward free-form answering, and supports both outcome- and process-level evaluation.

  • Benchmark scope: 1.2K samples span 12 task categories, with 100 samples per category, organized across 2D structured, 3D scene, and real-world visual reasoning.The categories cover six 2D structured tasks, two 3D scene tasks, and four real-world visual reasoning tasks.
  • Task design: The 2D, 3D, and real-world levels require operations such as marking relations, isolating regions, tracking changes, and filtering distractors.These operations target geometric and symbolic evidence, object attributes and spatial relations, or perceptual and state changes.
  • Construction pipeline: Candidate samples are filtered for visual dependency, normalized into a unified answer format, and checked automatically and manually for answer uniqueness and image–question consistency.Caption-solvable samples are removed, while retained problems are manually rechecked for visual dependency, ambiguity, and compatibility with visual-action reasoning.
  • Answer-format conversion: Whenever possible, multiple-choice or template-based questions are rewritten as free-form queries to reduce guessing shortcuts and answer-option priors.127 of 1,200 samples retain structured candidate sets because the candidates themselves form part of the evidence required for answering.
  • Evaluation targets: See2ThinkBench evaluates both final-answer correctness and process-level quality, including action relevance, render faithfulness, and feedback uptake.Final answers use exact matching or semantic equivalence, while complete VAoT trajectories are assessed on their key visual steps.

3 Visual Action-of-Thought

Visual Action-of-Thought (VAoT) is an inference-time protocol that records interleaved textual thoughts, grounded visual actions, and rendered states. Its externally executable closed loop enables controlled intervention and process-level diagnosis without model-internal access.

  • Protocol: VAoT records textual thoughts, image-grounded actions, and rendered visual states as an interleaved reasoning trajectory.Because each action can be executed and inspected externally, the trajectory supports controlled intervention and process-level diagnosis.
  • Protocol: At each step, the model emits a textual thought and structured action, an external renderer executes it, and the updated state returns for further reasoning.The action specifies an operation, normalized target coordinates, and optional semantic content or style.
  • Visual actions: The action space covers highlighting, auxiliary lines, cropping or zooming, and visual-relation annotation over visible or visually grounded content.Each operation is executable and externally inspectable because it must target visible image content or a visually grounded construction.
  • Renderer: The renderer is a constrained external workspace that performs requested edits while preserving unrelated content, rather than solving the task.It receives neither the reference answer nor a reasoning objective.
  • Protocol: VAoT requires no access to model internals, making it an inference-time diagnostic protocol.Execution and intervention conditions are defined separately in Section 4.1.

4 Experiments

Experiments across the complete 1.2K-sample benchmark show that visual reasoning is conditional: its strongest setting varies by model and task regime rather than consistently improving accuracy. Process analyses identify rendering fidelity as the main bottleneck, while corrupted feedback reveals behavioral dependence on returned visual states.

  • Outcome results: No single inference setting dominates: GPT-5.5 and Qwen3-VL-32B-Instruct perform best with CoT, GPT-o3 with VAoT, and Gemini 3.5 Flash with VAoT-NoRender.The effect of intermediate visual reasoning depends on both the model and the task.
  • Outcome results: Robot Manipulation is a consistent low-accuracy outlier because precise target grounding and the immediate manipulation command leave little room for visual actions to compensate for errors.Nearby-object selection, scene description, or a multi-step plan is counted as incorrect.
  • Task and model regimes: Structured 2D problems favor direct reasoning, VAoT-NoRender leads in 3D scenes, and real-world strategies are nearly tied without a clear aggregate winner.Model profiles also differ: GPT-5.5 and Qwen3-VL-32B-Instruct are text-first, Gemini 3.5 Flash planner-first, and GPT-o3 slightly favors closed-loop reasoning.
  • Process diagnosis: Action Relevance is near saturation, whereas Render Faithfulness is substantially lower across models and task groups, making accurate rendering the clearest process bottleneck.Qwen3-VL-32B-Instruct has the highest overall Feedback Uptake (0.871) despite lower final-answer accuracy, showing that feedback use is distinct from successful reasoning.
  • Rendering effects: −9.3 percentage points is the transition balance for completely unfaithful rendering in 3D reasoning, while higher Render Faithfulness primarily reduces render harm.The corresponding balances are −6.1 percentage points in 2D and 2.2 percentage points in real-world reasoning.
  • Feedback dependence: 32.7–54.2% of samples change semantic answer under corrupted feedback across the four models, even when the net accuracy effect is smaller.Feedback Uptake tracks corrupted-feedback sensitivity most clearly in 3D reasoning, but the association remains environment-dependent.

5 Related Work

Prior work extends multimodal chain-of-thought with active visual tools and introduces benchmarks for visual-interleaved reasoning trajectories. However, existing evaluations do not jointly establish whether visual actions are task-relevant, faithfully rendered, and behaviorally used, motivating more comprehensive diagnosis.

  • Multimodal chain-of-thought and active visual reasoning: Multimodal chain-of-thought evaluations assess rationale quality, robustness, efficiency, and error localization, while active visual reasoning manipulates task-relevant image regions through tools.Methods include cropping, zooming, annotating, and sketching.
  • Benchmarks for visual intermediate states: Existing benchmarks study intermediate visualizations, free-form visual-interleaved trajectories, process reward modeling, and dynamically generated future-frame reasoning.Examples include MIRA, ViC-Bench, TWI-PRMBench, and TwiFF-Bench.
  • Benchmarks for visual intermediate states: These benchmarks do not jointly diagnose whether a self-generated visual action is task-relevant, faithfully realized in the returned state, and behaviorally used later.The missing diagnosis spans action relevance, rendering fidelity, and dependence on returned visual evidence.
  • Diagnosing visual-state use: Recent work evaluates whether intermediate visual information is faithful or answer-critical through consistency checks, restricted information flow, and comparisons of tool-based methods.View Dropout restricts direct information flow from original views to the final answer to encourage reliance on generated thinking images.

6 Conclusion and Limitations · Appendix · Appendix Contents

See2Think introduces a unified benchmark and intervention protocol for studying whether multimodal models genuinely use intermediate visual states. Its limitations concern model coverage, external rendering and automatic evaluation, while the appendices document construction, inference settings, evaluation protocols, complete results, validation, and qualitative trajectories.

  • 6 Conclusion and Limitations: See2Think combines See2ThinkBench with VAoT to study multimodal models’ use of intermediate visual states during reasoning.VAoT records visual actions, rendered states, and subsequent reasoning.
  • 6 Conclusion and Limitations: The evaluation covers four representative multimodal models and may not capture the full diversity of current and future vision-language systems.The limitation concerns model coverage rather than a claim about all vision-language systems.
  • 6 Conclusion and Limitations: VAoT depends on an external rendering component and automatic process evaluation, which may introduce errors despite human validation.The measured effect of VAoT-WrongRender also depends on intervention quality.
  • Appendix Contents: The benchmark-construction appendix details source-dataset collection, preprocessing, caption-only visual-dependency filtering, answer-format conversion, and manual quality checking.These topics appear under sections A, A.1, A.2, A.3, and A.4.
  • Appendix Contents: The inference appendix covers implementation settings, Text-CoT and VAoT prompts, VAoT-NoRender, and the WrongRender intervention prompt.These topics appear under sections B, B.1, B.2, B.3, B.4, and B.5.
  • Appendix Contents: The remaining appendices provide final-answer, process-level, and paired-intervention protocols; complete 1.2K outcomes and intervention statistics; process statistics; human validation; WrongRender audits; and qualitative trajectories.The qualitative materials include successful visual-state use, process-level failure cases, and answer changes under WrongRender.

A Benchmark Construction Details … B Inference Prompts and Implementation Details

See2ThinkBench is constructed through source aggregation, visual-dependency filtering, answer-format conversion, and manual quality checks, while inference prompts impose explicit text-only or visual-action reasoning protocols. The prompts distinguish rendered, non-rendered, and intentionally corrupted visual states to control how models generate and use intermediate evidence.

  • A Benchmark Construction Details: The benchmark construction pipeline comprises source collection, caption-only visual-dependency filtering, answer-format conversion, and manual quality checking.These stages are applied to build See2ThinkBench from heterogeneous visual reasoning sources.
  • A.1 Source Dataset Collection and Preprocessing: Samples are collected from nine datasets, mapped into unified 2D structured, 3D scene, and real-world visual reasoning categories.Sources include RBench-V, EMMA, M3CoT, PRISM-Bench, CLEVR, SuperCLEVR, VLABench, DROID, and IntPhys2; invalid or visually insufficient samples are removed.
  • A.2 Caption-Only Visual-Dependency Filtering: Caption-only filtering removes samples solved correctly in at least three of five question-and-caption trials, retaining problems requiring direct visual access.The procedure reduces text-dominant samples without claiming retained problems are impossible from language priors.
  • A.3 Answer-Format Conversion: Free-form conversion removes answer-choice lists while preserving visual evidence and task objectives, but 127 of 1,200 samples retain structured candidate sets.Converted questions must have unique, reproducible reference answers without introducing information or revealing the correct option.
  • A.4 Manual Quality Checking: Manual reviewers verify image grounding, uniquely supported answers, sufficient visual evidence, and meaningful operations such as highlighting, cropping, drawing lines, or annotating relations.Ambiguous, underspecified, inconsistent, or text-shortcut samples are revised or removed.
  • B.2 Text-CoT Prompt: The Text-CoT prompt requires complete step-by-step solutions whose reasoning explicitly references visual evidence and would be incomplete without the image.Each step includes a visual observation, reasoning, and calculation or result, followed by a final answer.
  • B.3 VAoT Prompt: The VAoT prompt makes visual actions conditional on unique explanatory necessity, uses atomic deductions, and generally limits solutions to 1-3 visual actions.Step 1 always contains exactly one visual action, whereas later steps default to text-only unless a new region, object, path, or spatial relation requires visualization.
  • B.4 VAoT-NoRender Prompt and B.5 WrongRender Intervention Prompt: VAoT-NoRender permits proposed actions but forbids treating them as evidence, while VAoT-WrongRender corrupts rendered states under the modify_key condition.NoRender actions are not executed; WrongRender modifies task-relevant image areas to create misleading problem-solving information.

B.1 Implementation Settings · C Detailed Evaluation Protocols · C.1 Final-Answer Evaluation

The evaluation uses four models, four controlled inference settings, and the complete 1,200-sample benchmark, with process analysis covering 4,800 VAoT trajectories. Final answers are scored by exact matching or semantic judging under explicit normalization and correctness rules.

  • B.1 Implementation Settings: Four models are evaluated across all four inference settings using the same 1,200 samples, original images, answer format, and interaction budget.Process scores cover 1.2K VAoT trajectories per model, totaling 4,800 trajectories; GPT-5.4 serves as the external answer and process judge.
  • B.1 Implementation Settings: The benchmark reports category-level outcomes and Figure 5 over all 1,200 samples, while group-level aggregates appear in Appendix D.These reporting choices accompany the common evaluation setup across models and inference settings.
  • B.1 Implementation Settings: The renderer receives only the current image state and structured action, executes the requested operation, and preserves unrelated content.VAoT-WrongRender changes only the returned visual state, while main interventions use modify_key to alter task-relevant evidence while preserving natural appearance and broad operation.
  • C.1 Final-Answer Evaluation: Final structured answers are scored by exact matching after deterministic normalization of case, whitespace, punctuation, units, and common numerical forms.Free-form predictions requiring semantic comparison are judged against the reference answer.
  • C.1 Final-Answer Evaluation: Semantic answer evaluation compares the model final answer with the reference answer for correctness, using the question, reference answer, and model answer as inputs.The evaluator is prompted to act as a strict evaluator for a multimodal reasoning benchmark and return a JSON object containing correctness and a brief reason.
  • C.1 Final-Answer Evaluation: The rules accept harmless formatting differences, equivalent wording, algebraically or numerically equivalent math, and either the correct multiple-choice letter or text.Contradictory extra claims are incorrect, and empty, evasive, or non-answer responses are not accepted.
  • C.1 Final-Answer Evaluation: The final-answer protocol returns only a JSON object with a boolean correctness field and a brief explanation.This output format is specified as {"correct": true or false, "reason": "brief explanation"}.

C.2 Process-Level Evaluation · C.3 Paired Intervention Analysis · D Complete Outcome and Paired-Intervention Results

The paper evaluates VAoT trajectories by identifying a key visual step and scoring action relevance, render faithfulness, and feedback uptake. It also uses paired interventions to separate rendering benefits or harms and measure accuracy degradation from corrupted visual feedback.

  • C.2 Process-Level Evaluation: The evaluator receives the question, reference answer, model final answer, and complete trajectory while withholding model identity.The final response is included to assess whether rendered-state information is incorporated into later reasoning and the answer.
  • C.2 Process-Level Evaluation: VAoT trajectories are evaluated by selecting the single visual step most relevant to final reasoning or most directly exposing failure.If no effective visual operation exists, the evaluator returns null.
  • C.2 Process-Level Evaluation: Action Relevance measures whether visual actions target task-relevant evidence and support solving the question.Scores are 0, 0.5, or 1, ranging from irrelevant or decorative actions to directly useful ones.
  • C.2 Process-Level Evaluation: Render Faithfulness measures whether rendered visual states faithfully execute intended visual actions.A score of 0 indicates missing, wrong, misleading, or inconsistent rendering; partial matches receive 0.5.
  • C.2 Process-Level Evaluation: Feedback Uptake measures whether subsequent reasoning uses rendered visual states as evidence rather than relying independently on text priors.Scores are 0, 0.5, or 1, depending on whether the rendered state is ignored, weakly used, or clearly used.
  • C.3 Paired Intervention Analysis: Paired analysis compares VAoT-NoRender → VAoT to separate rendering benefit from rendering harm.Transition shares are normalized by all matched trajectories within each Render Faithfulness group.
  • C.3 Paired Intervention Analysis: The second paired comparison, VAoT → VAoT-WrongRender, measures accuracy degradation under corrupted visual feedback.Trajectories are grouped by Feedback Uptake, and the reported quantity is AccVAoT − AccWrongRender.

D.1 Complete 1.2K Outcome Results … F.1 Human Validation of Process Annotations

The supplementary results consolidate complete-benchmark outcomes, paired intervention statistics, and process-score analyses, then validate the process annotations through a structured human audit. The audit covers 240 trajectories across four models and three task groups, with both annotators judging more than 92% of every target at least partially reasonable.

  • D.1 Complete 1.2K Outcome Results: Table 5 summarizes accuracy across all four settings on the complete 1,200-sample benchmark by visual-reasoning environment.The main paper reports corresponding results across 12 categories on the same samples.
  • D.2 Complete Paired-Intervention Statistics: Tables 6–8 report rendering benefit and harm, sensitivity to corrupted feedback, and semantic answer changes using complete aligned records from all four models.Benefit and harm are computed within task group and Render Faithfulness bin, while Accuracy Drop is AccVAoT − AccWrongRender.
  • E Complete Process Statistics: Table 9 splits Action Relevance, Render Faithfulness, and Feedback Uptake process scores by final-answer correctness across the complete four-model benchmark.These counts underlie the correct-minus-incorrect gaps reported in Figure 6(b).
  • F.1 Human Validation of Process Annotations: The human validation audit independently reviews the same 240 VAoT trajectories sampled across four models, three task groups, four diagnostic strata, and five trajectories per cell.The strata cover high-action/failed-answer cases, rendering failures, partial renders, and weak action selection.
  • F.1 Human Validation of Process Annotations: Annotators assess key-step selection and three process scores against the original problem, complete trajectory, selected step, and judge explanations, labeling each target reasonable, partially reasonable, or unreasonable.Feedback Uptake receives credit only for information actually present in the rendered state.
  • F.1 Human Validation of Process Annotations: More than 92% of judgments for every target are at least partially reasonable across both annotators.The largest disagreements occur at the strict Reasonable-versus-Partially-Reasonable boundary for Action Relevance and Render Faithfulness.

F.2 WrongRender Quality Audit · G Qualitative Trajectories · G.1 Successful Visual-State Use

The WrongRender audit finds that corrupted visual states are often plausible and task-relevant but imperfect interventions, while paired analyses still show behavioral dependence on visual feedback. Successful trajectories demonstrate that models can select, faithfully render, and use task-relevant visual actions across 2D, 3D, and real-world reasoning.

  • F.2 WrongRender Quality Audit: F.2 WrongRender Quality Audit: 120 audited cases were evaluated for corruption validity, plausibility, operation consistency, task relevance, and content preservation.Strict Pass requires no failed criteria; Acceptable allows at most one failure.
  • F.2 WrongRender Quality Audit: F.2 WrongRender Quality Audit: 56.7% of the combined 120-case audit achieved Strict Pass, while 78.3% were Acceptable.The audit treats WrongRender as an aggregate diagnostic because individual corrupted states may not realize the intended modification equally well.
  • F.2 WrongRender Quality Audit: F.2 WrongRender Quality Audit: 8.82 percentage points was the combined four-model accuracy drop on the strict subset, while the relaxed subset retained a 3.19-point drop.The strict subset excludes any failed audit criterion; the relaxed subset allows at most one.
  • F.2 WrongRender Quality Audit: F.2 WrongRender Quality Audit: Qwen-only strict and acceptable subsets showed accuracy drops of 8.70 and 7.69 percentage points, respectively.The behavioral-dependence result is not restricted to failed interventions, although its estimated magnitude is sensitive to intervention quality.
  • G Qualitative Trajectories: G Qualitative Trajectories: Representative trajectories span 2D structured, 3D scene, and real-world visual reasoning, including successful use, process failures, and answer changes under corrupted feedback.The panels use original model-produced images without redrawing.
  • G.1 Successful Visual-State Use: G.1 Successful Visual-State Use: Successful cases involve task-relevant actions that are faithfully rendered and explicitly used in subsequent reasoning, receiving scores of 1 on all three process dimensions.The dimensions are Action Relevance, Render Faithfulness, and Feedback Uptake.
  • G.1 Successful Visual-State Use: G.1 Successful Visual-State Use: Examples include highlighting parallel resistor branches, marking a target bus for relation-based counting, and grounding a woman and laptop for activity recognition.These examples cover 2D structured, 3D scene, and real-world reasoning, respectively.

G.2 Process-level Failure Cases · G.3 Answer Changes under WrongRender

The failure cases show that proposing a relevant visual action does not ensure faithful rendering or reliable feedback use. WrongRender interventions preserve operation intent but alter task-relevant content, changing subsequent reasoning or final answers across environments.

  • G.2 Process-level Failure Cases: Relevant action proposals can fail at execution and use: the 2D render is partial and partially used, scored 1, 0.5, 0.5.This separates action relevance, rendering faithfulness, and feedback uptake.
  • G.2 Process-level Failure Cases: The 3D case marks the wrong object and is not used reliably, scored 1, 0, 0.The action is relevant, but rendering and subsequent use both fail.
  • G.2 Process-level Failure Cases: The robot-manipulation case is only partially faithful and receives no measurable uptake, scored 1, 0.5, 0, causing an incorrect continuous action prediction.The failure illustrates how execution and feedback-use errors can jointly affect the final prediction.
  • G.2 Process-level Failure Cases: Figure 10 separates relevant action proposal from execution quality and subsequent use, showing why high Action Relevance alone does not guarantee a correct answer.The representative cases span 2D, 3D, and robot-manipulation reasoning.
  • G.3 Answer Changes under WrongRender: WrongRender interventions preserve broad operation intent while corrupting task-relevant content: prism geometry, target relations, or tie branding.The comparison covers one case from each environment.
  • G.3 Answer Changes under WrongRender: Corrupted visual states change subsequent reasoning or final answers in every illustrated environment.The interventions affect 2D structured, 3D scene, and real-world reasoning cases.
  • G.3 Answer Changes under WrongRender: In the representative 2D case, corrupted prism geometry changes the inferred refractive index.This is a qualitative WrongRender illustration rather than an aggregate intervention-quality result.
  • G.3 Answer Changes under WrongRender: In the 3D case, corrupted relation markers change the object count from 2 to 5, while an unbranded tie changes a real-world interpretation from promotional display to decorative mannequin.Figure 11 presents these as qualitative examples; aggregate intervention quality is assessed separately by the human quality audit in Appendix F.2.
Loading 2607.26769v1…