PaperlayerView exact revision on arXiv

Source-linked AI summary

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

arXiv:2607.26769v1cs.CVcs.AI

TL;DR

Whether multimodal models genuinely rely on intermediate visual states remains unclear, and existing evaluations do not jointly assess how those states are generated and used. See2Think pairs a visually dependent benchmark with a protocol that records visual actions, rendered states, and subsequent reasoning; experiments find that no inference setting dominates across tasks.

  • Problem

    Existing evaluations do not jointly test whether self-generated visual actions are task-relevant, faithfully rendered, and behaviorally used in subsequent reasoning.

    Source: §1 · p. 2

  • Method

    See2Think pairs a 1.2K-sample benchmark of visually dependent problems across 12 task categories with VAoT, which records visual actions, rendered states, and subsequent reasoning.

    Source: §1 · p. 2

  • Results

    No inference setting dominates across tasks: the strongest setting varies by model, and intermediate visual reasoning effects depend on both model and task.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Models’ generally relevant visual-operation choices do not eliminate faithful rendering and use of returned visual evidence as bottlenecks.

    Source: §6 · p. 12

    The evaluation covers four representative multimodal models and may not capture the full diversity of current and future vision-language systems.

    Source: §6 · p. 12

Abstract

from arXiv · show

Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.

1 Introduction

See2Think evaluates whether multimodal models genuinely use intermediate visual states, addressing gaps in benchmarks that do not jointly diagnose action relevance, rendering faithfulness, and behavioral dependence. It combines a visually dependent benchmark with an observable, intervenable reasoning protocol.

Motivation

  • Existing evaluations often emphasize final answers [17] [26], textual rationales [8], oracle clues, or aggregate process scores, leaving visual-state use jointly undiagnosed.Task collections may also lack visual diversity or allow text-only shortcuts.Source: §1 · p. 2, Figure 1 · p. 2

See2Think framework

  • See2Think jointly tests whether visual actions are relevant, rendered states are faithful, and subsequent reasoning depends on returned visual evidence.This controlled diagnostic distinguishes whether a visual state is useful from whether model behavior actually depends on it.Source: §1 · pp. 2–3, Figure 1 · p. 2

Visual Action-of-Thought (VAoT)

  • VAoT records alternating textual thoughts, visual actions, rendered states, and subsequent reasoning across four controlled inference settings.The settings compare CoT, action planning without rendering, closed-loop VAoT, and task-relevant corrupted feedback.Source: §1 · pp. 2–3

See2ThinkBench

  • See2ThinkBench contains 1.2K visually dependent problems across 12 categories spanning 2D structured, 3D scene, and real-world reasoning.Caption-only solvability filtering reduces text-dominant shortcuts, and answer-format normalization supports consistent evaluation of free-form and structured outputs.Source: §1 · pp. 2–3

Experimental overview

  • Experiments across four representative proprietary and open-source models find no universally optimal visual-reasoning strategy and identify a persistent bottleneck after action selection.The evaluations also examine when rendered feedback helps, harms, or merely decorates reasoning.Source: §1 · p. 3

2 See2ThinkBench

See2ThinkBench is designed to diagnose intermediate visual-state reasoning at both outcome and process levels, using problems that require visual evidence, support meaningful visual operations, and permit matched evaluation.

2.1 Task Design and Data Sources

  • See2ThinkBench contains 1,200 samples across 12 task categories—100 per category—spanning 2D structured, 3D scene, and real-world visual reasoning.The categories include geometry, spatial puzzles, physics, chemistry, science QA, abstract patterns, object attributes, compositional 3D, robot tasks, visual commonsense, and intuitive physics; useful operations range from drawing and annotation to isolating evidence and tracking changes.Source: §2.1 · pp. 3–4
  • See2Think combines text-shortcut filtering with self-generated closed-loop visual states, stage-wise diagnosis of actions, rendering, and feedback use, and corrupted-feedback interventions.These features distinguish it from representative benchmarks that focus on annotated intermediate visualizations, free-form visual-interleaved trajectories, process reward modeling, or dynamically generated future states.Source: §2.1 · p. 4

2.2 Data Construction

  • Caption-only filtering removes samples answerable in at least three of five trials, retaining candidates that are more visually dependent.A multimodal captioner describes observable evidence without answering; a reasoning model then receives the question and caption but not the original image.Source: §2.2 · pp. 4–5
  • Answer options are removed when possible: 1,073 samples use free-form answers, while 127 retain structured candidate sets that are part of the reasoning problem.Rewritten samples undergo automatic and manual checks for visual dependency, answer uniqueness, ambiguity, and image–question consistency.Source: §2.2 · pp. 4–5

2.3 Evaluation Targets

  • The benchmark evaluates reference-answer correctness and trajectory quality through action relevance, render faithfulness, and feedback uptake.Structured answers use normalized exact matching, free-form answers use semantic equivalence, and process scoring targets the key visual steps most relevant to the final reasoning.Source: §2.3 · pp. 5–6

3 Visual Action-of-Thought

Visual Action-of-Thought (VAoT) is an inference-time protocol that records interleaved textual thoughts, image-grounded actions, and rendered visual states. Its externally inspectable trajectories enable controlled intervention and process-level diagnosis without access to model internals.

3 Visual Action-of-Thought

  • VAoT records a closed-loop sequence of textual thoughts, visual actions, and externally rendered states, enabling controlled intervention and process-level diagnosis.The protocol requires no access to model internals.Source: §3 · pp. 6–7
  • For each input image and question, the model specifies a thought and structured action; an external renderer executes the action and returns an updated state until a final answer.Actions specify an operation, normalized target coordinates, and optional semantic content or style; they can highlight, draw, crop or zoom, or annotate visually grounded relations.Source: §3 · p. 6
  • The renderer is a constrained external workspace, not a task-solving agent, and receives neither the reference answer nor a reasoning objective.The execution and intervention conditions are defined in Section 4.1.Source: §3 · pp. 6–7

4 Experiments

Across the benchmark, no inference setting consistently dominates across models and tasks. Process analyses identify rendering as a key bottleneck and show that sensitivity to visual feedback depends on the environment.

4.1 Experimental Setup

  • The experiments compare four models across four inference settings on the complete 1.2K-sample benchmark.The settings are CoT, VAoT-NoRender, VAoT, and VAoT-WrongRender; answers are evaluated by normalized exact matching or semantic equivalence, and VAoT trajectories receive three process scores.Source: §4.1 · p. 7

4.2 Main Results

  • No inference setting dominates: GPT-5.5 and Qwen3-VL-32B-Instruct perform best with CoT, GPT-o3 with VAoT, and Gemini 3.5 Flash with VAoT-NoRender.These category-level results cover the complete 1.2K-sample benchmark; the effect of intermediate visual reasoning varies by model and task.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
  • Robot Manipulation is a consistent low-accuracy outlier because precise target grounding and an immediate command are required.Nearby-object selection, scene description, or multi-step plans count as incorrect, leaving little room for visual actions to compensate for grounding errors.Source: §4.2 · p. 8

4.3 When Does Visual Thinking Help?

  • Which strategy helps varies by task regime: direct reasoning favors structured 2D problems, VAoT-NoRender is strongest on average in 3D, and three strategies nearly tie in real-world scenes.GPT-5.5 and Qwen3-VL-32B-Instruct have text-first profiles, Gemini 3.5 Flash a planner-first profile, and GPT-o3 a slight closed-loop preference.Source: §4.3 · p. 8

4.4 Explaining Outcome Patterns through Process-level Diagnosis

  • Action Relevance is near saturation, but Render Faithfulness is substantially lower, making faithful execution the clearest process bottleneck.Qwen3-VL-32B-Instruct has the highest overall Feedback Uptake (0.871) despite lower final-answer accuracy, showing that using returned information is distinct from using it correctly.Source: §4.4 · pp. 9–10
  • At zero Render Faithfulness, transition balance is −6.1 percentage points in 2D, −9.3 in 3D, and 2.2 in real-world reasoning; at score 1, it is −1.2, 0.3, and 0.9, respectively.Higher Render Faithfulness primarily reduces render harm.Source: Figure 7 · p. 10, §4.4 · p. 10
  • As Feedback Uptake rises from 0 to 0.5 and 1, overall accuracy degradation changes from −2.3 to 3.4 and 3.1 percentage points.In 3D, the drop grows from 3.5 to 10.3 and 15.5 points; the association is weaker or non-monotonic in 2D and real-world tasks. Corrupted feedback changes the semantic answer in 32.7–54.2% of samples across the four models, even when its net accuracy effect is smaller.Source: Figure 8 · p. 10, §4.4 · pp. 10–11

4.5 Human Validation of Process-level Evaluation

  • Human review found 92.9–96.9% of judgments at least partially reasonable across the four process-evaluation targets.Two annotators assessed 240 trajectories; the larger strict-to-partial gaps for Action Relevance and Render Faithfulness indicate disagreement mainly at the boundary between fully and partially supported judgments.Source: §4.5 · p. 11, Table 4 · p. 11

5 Related Work

Prior work explores active visual reasoning, benchmarks intermediate visual traces, and investigates whether models use visual states. See2Think complements these efforts with an inference-time framework that separates visual-action relevance, rendering faithfulness, feedback uptake, and behavioral dependence.

Multimodal chain-of-thought and active visual reasoning

  • Active visual reasoning extends multimodal chain-of-thought through image tools [7] [21] [30], generated or interleaved visual states [12] [15] [19] [20], continuous actions [6] [29], and reliability methods [11] [25].Multimodal chain-of-thought evaluations also cover rationale quality, robustness, efficiency, and error localization [8] [28]; performance gains alone do not establish faithful rendering or subsequent use of a visual state.Source: §5 · p. 11

Benchmarks for visual intermediate states

  • MIRA and ViC-Bench [23] [31], TWI-PRMBench [14], and TwiFF-Bench [32] broaden evaluation, but do not jointly test action relevance, render faithfulness, and behavioral dependence on returned evidence.These benchmarks primarily measure the usefulness, plausibility, or overall quality of visual traces.Source: §5 · pp. 11–12

Diagnosing visual-state use

  • Prior studies assess visual-thought consistency [16], encourage use of generated images through View Dropout [24], or compare tool-based and tool-free agents [3]; See2Think offers a complementary, model-agnostic inference-time framework.It records self-generated visual actions, externally rendered states, and subsequent reasoning, using matched standard and corrupted-render conditions to distinguish action relevance, render faithfulness, feedback uptake, visual-state utility, and behavioral dependence.Source: §5 · p. 12

6 Conclusion and Limitations · Appendix · Appendix Contents

See2Think combines a visually dependent benchmark with an observable, intervenable protocol to assess how multimodal models generate and use intermediate visual states. Its findings identify model- and environment-dependent performance, rendering and evidence-use bottlenecks, and limitations in model coverage and process evaluation.

6 Conclusion and Limitations

  • See2Think combines See2ThinkBench’s visually dependent reasoning problems with VAoT’s record of visual actions, rendered states, and subsequent reasoning.The framework evaluates benchmark problems through a unified answer-generation interface and makes visual reasoning processes observable and intervenable.Source: §6 · p. 12
  • Across four evaluated models, visual reasoning was model- and environment-dependent, with no single universally effective strategy.The models were GPT-5.5, GPT-o3, Gemini 3.5 Flash, and Qwen3-VL-32B-Instruct.Source: §6 · p. 12
  • Models generally identified relevant visual operations, but faithfully rendering them and using returned visual evidence remained important bottlenecks.Source: §6 · p. 12
  • Visual-state utility and behavioral dependence were distinct: corrupted visual states could affect decisions even when the states offered limited direct benefit.Source: §6 · p. 12
  • The evaluation covered four representative multimodal models, so it may not capture the diversity of current and future vision-language systems.Source: §6 · p. 12
  • VAoT depends on an external renderer and automatic process evaluation, both of which may introduce errors despite human validation.Source: §6 · p. 12
  • The accuracy-drop magnitude for VAoT-WrongRender varied with intervention quality and filtering criteria, although quality-filtered human-audited analyses preserved its direction.Future work may broaden model coverage and explore learned rendering mechanisms and finer-grained process supervision.Source: §6 · p. 12

A Benchmark Construction Details … B Inference Prompts and Implementation Details

This section details See2ThinkBench construction and the prompts and inference variants used to evaluate text-only and visual-action reasoning, including standard VAoT, no-render, and wrong-render settings. It emphasizes visual dependence, answer integrity, grounding in image evidence, and the intervention that corrupts task-relevant visual evidence.

A Benchmark Construction Details

  • See2ThinkBench construction proceeds through source collection, caption-only visual-dependency filtering, answer-format conversion, and manual quality checking.The benchmark draws from multiple datasets and organizes tasks across 2D structured, 3D scene, and real-world visual reasoning; unusable or visually insufficient samples are removed.Source: A Benchmark Construction Details · p. 16, §A.1 · p. 16

A.1 Source Dataset Collection and Preprocessing

  • Source samples are mapped to See2ThinkBench categories, retain original images where available, and are screened for missing images, duplicates, invalid references, corruption, or insufficient visual evidence.The categories cover six 2D task types, two 3D task types, and four real-world task types drawn from the listed source datasets.Source: §A.1 · p. 16

A.2 Caption-Only Visual-Dependency Filtering

  • Caption-only filtering removes samples solved correctly in at least three of five question-and-caption trials, reducing text-dominant cases without claiming retained items defeat language priors.Captions describe observable evidence without answering or unsupported inference, and the original image is withheld from the solvability test.Source: §A.2 · p. 16

A.3 Answer-Format Conversion

  • Of 1,200 samples, 1,073 use free-form answers and 127 retain structured candidate sets when the alternatives themselves are part of the reasoning problem.Free-form rewrites must preserve the visual evidence and task objective while ensuring a unique, reproducible reference answer.Source: §A.3 · p. 17

A.4 Manual Quality Checking

  • Manual reviewers verify image grounding, uniquely supported reference answers, sufficient visual evidence, and natural opportunities for meaningful visual operations.Ambiguous references, underspecified answers, inconsistent image–question pairs, and substantial text-only shortcuts prompt revision or removal.Source: §A.4 · p. 18

B.2 Text-CoT Prompt

  • The Text-CoT prompt requires complete step-by-step solutions that explicitly reference image evidence and would be incomplete if the image were removed.Each step includes visual observation, reasoning, and calculations or intermediate results, followed by a final answer.Source: §B.2 · pp. 18–19

B.3 VAoT Prompt

  • The VAoT prompt asks for atomic deductions and visual actions only when they uniquely enable understanding beyond text, with later steps defaulting to text alone.Step 1 requires exactly one visual action; subsequent actions are reserved for new visual information, and the prompt recommends 1–3 actions across a solution.Source: §B.3 · pp. 19–20
  • VAoT prompts the model to generate one next reasoning step from the problem context and prior steps, continuing until it can provide a final answer.The supplied prompt excerpt includes the task instruction and context fields, along with a final-answer format.Source: §B.3 · p. 21

B.4 VAoT-NoRender Prompt

  • VAoT-NoRender allows proposed visual actions but does not render them, so action suggestions cannot be treated as visual evidence.The prompt limits reasoning to the problem, original image, and prior text steps; actions are reserved for locating regions, tracing paths, showing spatial relations, separating objects, or zooming into detail.Source: §B.4 · pp. 21–22

B.5 WrongRender Intervention Prompt

  • VAoT-WrongRender preserves standard VAoT’s reasoning procedure but corrupts the rendered state; experiments use modify_key, which alters task-relevant evidence, not modify_non_key.The supported modify_non_key condition changes secondary details while keeping important information correct, but is not used in the reported experiments.Source: §B.5 · pp. 22–23

B.1 Implementation Settings · C Detailed Evaluation Protocols · C.1 Final-Answer Evaluation

The implementation evaluates four multimodal models under matched benchmark and interaction conditions, with controlled visual-feedback corruption for intervention analyses. Final answers are scored by normalized exact match or semantic judgment against references, using strict correctness rules.

B.1 Implementation Settings

  • All four models use each of the four inference settings on the same 1,200 samples, original images, answer format, and interaction budget; GPT-5.4 serves as external answer and process judge.The category-level outcome table and Figure 5 cover all 1,200 samples; process scoring covers 1,200 VAoT trajectories per model, or 4,800 total.Source: §B.1 · p. 18
  • The main WrongRender intervention changes only the returned visual state while keeping the VAoT reasoning prompt and interaction budget unchanged.The modify_key condition alters task-relevant evidence while aiming to preserve natural appearance and broad operation intent; the model is not told feedback was corrupted.Source: §B.1 · p. 18
  • Paired intervention analyses use complete sample-level alignments across all 1,200 samples for all four models.The renderer receives the current image state and structured action, and faithful rendering is constrained to execute the requested operation while preserving unrelated content.Source: §B.1 · p. 18

C.1 Final-Answer Evaluation

  • Structured answers are scored by exact match after deterministic normalization, while free-form predictions requiring semantic comparison are judged against the reference answer.Normalization covers case, whitespace, punctuation, unit formatting, and common numerical forms.Source: §C.1 · p. 24
  • Contradictory extra claims, empty or evasive answers, and errors in counts, directions, relations, or named entities are marked incorrect.The evaluation prompt requests a JSON object containing a correctness judgment and brief reason.Source: §C.1 · p. 24
  • Semantic evaluation accepts harmless formatting differences, equivalent wording, algebraically equivalent math, numerically equivalent answers, and either the correct multiple-choice letter or option text.The evaluator judges semantic correctness with respect to the reference answer.Source: §C.1 · p. 24

C.2 Process-Level Evaluation · C.3 Paired Intervention Analysis · D Complete Outcome and Paired-Intervention Results

The process evaluation selects a key visual step and scores action relevance, render faithfulness, and feedback uptake. Paired analyses compare rendering with no rendering and correct with corrupted visual feedback.

C.2 Process-Level Evaluation

  • The judge receives the question, reference answer, final answer, and full trajectory, while model identity and explicit correctness labels are withheld.The final answer is included so the judge can assess whether the rendered state informs subsequent reasoning and the response.Source: §C.2 · p. 24
  • The evaluator selects the visual step most relevant to final reasoning or most directly exposing a failure, using null when no effective visual operation exists.The selected step is then evaluated on three dimensions.Source: §C.2 · p. 25
  • Action Relevance scores whether visual actions target useful task evidence, while Render Faithfulness scores whether rendered states execute those actions accurately.Both dimensions use scores of 0, 0.5, or 1; render scores distinguish faithful execution from partial matches or missing, wrong, or misleading states.Source: §C.2 · p. 25
  • Feedback Uptake scores whether subsequent reasoning uses rendered states as evidence rather than ignoring them or relying mostly on text priors.The rubric assigns 0, 0.5, or 1 for absent, weak or implicit, or clear use of visual feedback.Source: §C.2 · p. 25

C.3 Paired Intervention Analysis

  • The VAoT-NoRender → VAoT comparison separates rendering benefit from rendering harm, with transition shares normalized within Render Faithfulness groups.Normalization uses all matched trajectories in each group.Source: §C.3 · p. 25
  • The VAoT → VAoT-WrongRender comparison measures accuracy degradation under corrupted visual feedback.Trajectories are grouped by Feedback Uptake, and the reported quantity is AccVAoT − AccWrongRender.Source: §C.3 · p. 25

D.1 Complete 1.2K Outcome Results … F.1 Human Validation of Process Annotations

The appendices provide complete-benchmark and paired-intervention summaries, process-score comparisons, and a human reliability check. The audit finds that both annotators judge more than 92% of targets at least partially reasonable, while identifying strict-boundary disagreements for two process dimensions.

D.1 Complete 1.2K Outcome Results

  • Table 5 summarizes accuracy for all four settings on the complete 1,200-sample benchmark by visual-reasoning environment.The main paper reports corresponding results across 12 categories on the same samples.Source: §D.1 · p. 26, Table 5 · p. 26

D.2 Complete Paired-Intervention Statistics

  • Tables 6–8 report paired statistics for four aligned models, covering rendering benefit and harm, accuracy sensitivity to corrupted feedback, and semantic answer changes.Semantic answer changes are determined by a judge rather than direct string comparison.Source: §D.2 · p. 26, Table 6 · p. 26, Table 7 · p. 27

E Complete Process Statistics

  • Table 9 splits Action Relevance, Render Faithfulness, and Feedback Uptake scores by final-answer correctness across the complete benchmark for four evaluated models.These scores are used to compute the correct-minus-incorrect gaps reported in Figure 6(b).Source: E Complete Process Statistics · p. 28, Table 9 · p. 28

F.1 Human Validation of Process Annotations

  • Two annotators independently review the same 240 VAoT trajectories sampled across four models, three task groups, four diagnostic strata, and five trajectories per cell.The sampling design is intended to cover high-action/failed-answer cases, rendering failures, partial renders, and weak action selection.Source: §F.1 · p. 28
  • Annotators inspect each trajectory and judge whether its selected key step and process scores are reasonable, partially reasonable, or unreasonable; Feedback Uptake credit requires information present in the rendered state.For each trajectory they review the image, question, full trajectory, selected key step, process scores, and judge explanations.Source: §F.1 · pp. 28–29
  • Both annotators mark more than 92% of judgments at least partially reasonable, with their greatest disagreement at the strict boundary for Action Relevance and Render Faithfulness.The pooled rates are reported in the main paper; Table 10 gives per-annotator summaries.Source: §F.1 · p. 29, Table 10 · p. 29

F.2 WrongRender Quality Audit · G Qualitative Trajectories · G.1 Successful Visual-State Use

The WrongRender audit finds behavioral dependence that persists in quality-filtered subsets, although its estimated magnitude depends on intervention quality. Representative trajectories also show successful visual-state use when actions are relevant, faithfully rendered, and taken up in reasoning.

F.2 WrongRender Quality Audit

  • The audit assessed 120 cases across four models and three task groups for corruption validity, plausibility, operation consistency, task relevance, and preservation of unrelated image content.The original 90 cases were sampled uniformly across GPT-5.5, GPT-o3, and Gemini 3.5 Flash, with 10 per model–group cell; 30 Qwen3-VL-32B-Instruct cases were added with the same task-group coverage.Source: §F.2 · p. 29
  • In the combined 120-case audit, 56.7% of cases were Strict Pass and 78.3% Acceptable; the added Qwen cases reached 76.7% and 86.7%, respectively.Strict Pass means no criterion failed; Acceptable allows at most one failed criterion.Source: §F.2 · p. 29, Table 11 · p. 30
  • WrongRender is treated as an aggregate diagnostic intervention, not as evidence that every corrupted state realizes the intended modification equally well.A case was inspected alongside the original image, requested action, correct rendered state, and WrongRender state, with each criterion rated Pass, Partial, or Fail.Source: §F.2 · pp. 29–30
  • Across all four models, WrongRender reduces accuracy by 8.82 points on the strict subset and 3.19 points on the relaxed subset; Qwen-only drops are 8.70 and 7.69 points, respectively.The positive drops show that behavioral dependence is not limited to interventions that fail the audit, but the estimated magnitude remains sensitive to intervention quality.Source: §F.2 · p. 30, Table 12 · p. 30

G Qualitative Trajectories

  • The qualitative trajectories span 2D structured, 3D scene, and real-world reasoning, illustrating successful visual-state use, process failures, and answer changes under corrupted feedback.Panels use original model-produced images without redrawing.Source: G Qualitative Trajectories · p. 31

G.1 Successful Visual-State Use

  • In Figure 9, successful trajectories score 1 for Action Relevance, Render Faithfulness, and Feedback Uptake, with examples across resistor networks, target-bus counting, and activity recognition.The examples show highlighting parallel branches for equivalent-resistance reasoning, marking a target bus for relation-based counting, and grounding a woman and laptop for activity recognition.Source: §G.1 · p. 31, Figure 9 · p. 31

G.2 Process-level Failure Cases · G.3 Answer Changes under WrongRender

The cases show that proposing a relevant visual action does not ensure faithful rendering or reliable use. WrongRender interventions preserve the broad operation intent but alter task-relevant content, changing subsequent reasoning or answers.

G.2 Process-level Failure Cases

  • The 2D case is partially rendered and used (1, 0.5, 0.5), while the robot-manipulation case is partially faithful with no measurable uptake (1, 0.5, 0), contributing to an incorrect action prediction.Source: §G.2 · pp. 31–32
  • In the 3D case, the model marks the wrong object and does not use the rendered state reliably (1, 0, 0).Source: §G.2 · p. 31
  • Relevant action proposals can still fail through inaccurate rendering or weak uptake, so Action Relevance alone does not guarantee a correct answer.Figure 10 illustrates failures in action execution and subsequent use.Source: Figure 10 · p. 32

G.3 Answer Changes under WrongRender

  • WrongRender preserves broad operation intent while corrupting task-relevant visual content, changing subsequent reasoning or the final answer across 2D, 3D, and real-world cases.Source: §G.3 · p. 32
  • In the 2D example, corrupted prism geometry changes the inferred refractive index.This is a qualitative illustration; aggregate intervention quality is assessed separately by a human quality audit.Source: Figure 11 · p. 33
  • In the 3D example, corrupted relation markers change the object count from 2 to 5.This is a qualitative illustration; aggregate intervention quality is assessed separately by a human quality audit.Source: Figure 11 · p. 33
  • In the real-world example, replacing a branded tie with an unbranded pattern changes its interpretation from a promotional display to a decorative mannequin.This is a qualitative illustration; aggregate intervention quality is assessed separately by a human quality audit.Source: Figure 11 · p. 33
Loading 2607.26769v1…