PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu

arXiv:2608.06270v1cs.AI

TL;DR

Visual tool-use can cost substantially more while delivering marginal gains, raising whether returned visual evidence actually affects answers. This paper causally audits tool-use across intervention levels and finds policy miscalibration: gains are concentrated in a calibrated minority, while many rollouts ignore evidence or use incoherent call schedules.

  • Problem

    Thinking-with-images often yields marginal or negative accuracy gains at higher token cost, motivating study of how returned visual evidence affects final predictions.

  • Method

    The paper models visual tool-use causally and applies policy-, trajectory-, and step-level interventions, including Visual Evidence Gain to isolate observation contributions.

  • Results

    Across six models and five benchmarks, policy miscalibration appears as Calling Without Looking and Looking Without Planning, with gains concentrated in a calibrated minority.

  • Takeaways & Limitations

    Aggregate accuracy gains can create an illusion of broadly effective tool-use because many rollouts either ignore returned evidence or schedule calls incoherently.

  • Takeaways & Limitations

    The conclusions are scoped by the study’s limited tool set.

Abstract

from arXiv · show

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.

1 Introduction

Thinking-with-images enables multimodal models to gather fine-grained visual evidence, yet often adds substantial token cost without improving accuracy and can even derail correct direct predictions. A causal audit identifies policy miscalibration—either observations are ignored causally or tool calls are poorly planned—and finds that gains come mainly from a calibrated minority of rollouts.

  • Thinking-with-images interleaves reasoning with crop-and-zoom operations to gather fine-grained evidence beyond a single holistic image view.
  • Tool-use often delivers marginal or negative accuracy improvements over direct inference despite substantially higher token cost.
  • Tool-augmented rollouts can repeatedly crop irrelevant regions and fail questions that direct prediction answers correctly.
  • A three-level causal intervention protocol compares policies, corrupts all trajectory observations, and counterfactually replaces individual observations to test their effects on answers.The protocol separates observation-mediated paths from action-induced shortcuts in visual tool-use.
  • Across six models and five fine-grained perception benchmarks, policy miscalibration appears as Calling Without Looking, where observations have no causal effect, or Looking Without Planning, where observations matter but calls are incoherent.The latter includes continuing after the answer is correct or repeatedly cropping uninformative regions.
  • Accuracy improvements are mainly driven by the Calibrated subset, while most tool calls are either decoupled from the answer or scheduled incoherently.The diagnostic also includes No-call rollouts and sorts every rollout into four groups.

2 Related Work

Related work spans active visual tool-use, process-aware training, and causal faithfulness analysis. This paper distinguishes prior claims about visual actions from an audit of whether returned observations causally mediate answers.

  • Thinking with images and visual tool-use: Active visual tool-use equips multimodal LLMs with operations such as crop-and-zoom, commonly attributing gains to higher-resolution evidence.Recent studies challenge this attribution, including a reported marginal-improvement paradox.
  • Thinking with images and visual tool-use: Wang et al. describe visual actions as potentially rationale-like but weakly grounded, whereas this work diagnoses aggregate gains that mask absent causal mediation on many trajectories.The distinction is between an “illusion of thinking with images” and this paper’s “illusion of visual tool-use.”
  • Thinking with images and visual tool-use: Process supervision can improve call quality but does not determine whether a call is needed, while ViRL averages about one visual action per example despite many cases being solvable by direct inference.The paper positions ViRL as a complementary training direction and its audit as a test of calibration and unnecessary or ungrounded calls.
  • Causal analysis and faithfulness of LLM reasoning: Causal faithfulness research tests whether intermediate reasoning artifacts causally support final answers, and this framework adapts Pearl interventions to multimodal tool-use traces.Visual Evidence Gain instantiates the natural indirect effect.

3 Causal Framework for Visual Tool-Use

The framework models visual tool-use as a causal system separating direct inference, observation-mediated evidence, and action-induced shortcuts. It audits these pathways through policy-, trajectory-, and step-level interventions, with Visual Evidence Gain isolating each observation’s causal contribution.

  • Step-level estimand: Visual Evidence Gain estimates the natural indirect effect along T_i → O_i → Y, while fixed-prefix counterfactuals cancel the action-induced shortcut.It therefore measures the local contribution of returned visual content rather than the mere presence of a tool call.
  • Causal model: A rollout SCM represents image and query inputs, tool actions, returned crops, and the final answer under one policy mechanism.At each step, the policy selects a crop-and-zoom action, the visual engine returns an observation, and the policy eventually produces an answer.
  • Causal model: The graph distinguishes a tool-free direct path, an intended observation-mediated path, and an action-induced shortcut through the call trace.The shortcut can shift answers merely because a call occurred, regardless of the returned visual content.
  • Interventions: Three interventions progressively test causal contribution: toggling tool-use policy, corrupting every rollout observation, and counterfactually replacing one observation under a fixed prefix.Trajectory corruption also captures downstream policy reactions, including altered actions, repair loops, stopping decisions, and budget exhaustion.

4 Diagnosing Policy Miscalibration

Policy miscalibration arises because tool-call timing and stopping decisions are decoupled from observations’ causal utility. The resulting diagnosis identifies two failure modes—Calling Without Looking and Looking Without Planning—while showing that positive policy-level gains are concentrated in the Calibrated subset.

  • Core finding: Policy miscalibration decouples decisions about calling, stopping, and committing from the causal utility of returned observations.Tools may be invoked when their evidence cannot help and continue after sufficient confidence is reached.
  • Failure modes: Calling Without Looking emits tool calls whose visual observations do not contribute to the answer.This occurs through saturated priors or structurally inactive calls.
  • Failure modes: Looking Without Planning produces non-zero visual evidence gains, but incoherent call schedules.The policy may continue after saturation or exhaust its tool-call budget.
  • Diagnostic classifier: The deterministic diagnostic partitions rollouts into No-call, Calling Without Looking, Calibrated, and Looking Without Planning using five vanilla-rollout features.The features are n, g0, peak per-call VEG, HitMax, and POER; thresholds are τsat = 0.95 and ϵ = 0.01.
  • ATE decomposition: Calibrated is the only group positive across all four evaluated models and accounts for most non-zero policy-level ATE.Mode 1 is small or offsetting, while Mode 2 and No-call remain within approximately ±1 percentage point; the same pattern holds on HR-Bench-4K.

5 Discussion

The discussion frames aggregate accuracy gains as an illusion of broadly causal visual tool-use because gains concentrate in calibrated trajectories while others ignore evidence or use incoherent schedules. It hypothesizes outcome-only RL as a common cause and proposes diagnostic, inference-time, and process-aware interventions.

  • The illusion of visual tool-use: Aggregate policy-level accuracy gains are real but concentrated in a calibrated minority, while many trajectories ignore returned evidence or use incoherent schedules.The paper names this discrepancy the illusion of visual tool-use.
  • An RL-trap hypothesis: Mode 1 and Mode 2 appearing across diverse models suggests outcome-only RL over tool-augmented rollouts as a plausible common factor.This remains a hypothesis rather than an established mechanism.
  • An RL-trap hypothesis: Outcome-only rewards reinforce shortcut behavior and fail to distinguish useful, redundant, and misleading calls, so matched training varying only the reward signal is needed.The paper leaves establishing this mechanism as future work.
  • Operational uses: The diagnostic evaluates interventions by requiring ∆fCal >0 and ∆fM1+∆fM2 <0, regardless of whether ATEpolicy moves.This is presented as an accuracy-independent, distributional evaluation criterion.
  • Operational uses: Inference-time adaptation can bypass Mode 1, stop Mode 2 before harmful late calls, and use step-level VEG to reward calibrated behavior while penalizing wasted or harmful calls.VEG supplies per-step supervision missing from outcome-only reward.

6 Conclusion

The paper reframes visual tool-use causally, separating observation-mediated paths from action-induced shortcuts and auditing them with interventions and Visual Evidence Gain. Across six models and five benchmarks, this protocol identifies policy miscalibration, including CWL and LWP, as the central bottleneck.

  • Causal audit: Visual tool-use is formulated as a causal graph partitioning trajectories into observation-mediated paths and action-induced shortcuts.This framing distinguishes visual evidence from effects attributable to the action itself.
  • Causal audit: A three-level intervention protocol and Visual Evidence Gain isolate the visual contribution of each call.The estimand is designed to measure per-call visual contribution.
  • Findings: Across six models and five benchmarks, the protocol exposes policy miscalibration as the central bottleneck, with failure modes CWL and LWP.A trajectory-level diagnostic additively decomposes the policy-level gain.

Limitations … A.2 Benchmarks and Answer Verification

The study’s conclusions are limited by access, tool coverage, and the unverified RL-trap hypothesis, while the appendix details model configurations, benchmark partitions, causal interventions, compute, and answer verification. Evaluation uses six public models, seven benchmark partitions, and a two-stage correctness pipeline combining normalized exact matching with judge-based assessment.

  • Limitations: The policy- and trajectory-level interventions require only model execution and observation control, whereas step-level Visual Evidence Gain requires token-level scores and white-box access.Whether the conclusions extend to closed-source models remains unresolved because models such as OpenAI o3/o4-mini expose neither token probabilities nor intermediate-observation intervention.
  • Limitations: The study examines only CROP-AND-ZOOM, so other tools may exhibit different calibration behaviors despite the framework’s proposed transfer through tool-specific counterfactual observations.Examples include segmentation, OCR, video frame selection, code-based image manipulation, and external search.
  • Limitations: The RL-trap explanation is a hypothesis rather than a causal finding, while the two failure modes and Calibrated-subset gain concentration hold independently of its confirmation.Controlled training studies varying rewards, rollout policy, or credit assignment while holding data and backbone fixed are left for future work.
  • A Experimental Setup Details: The appendix documents six thinking-with-images policies, seven evaluation benchmarks, three causal interventions, and the aggregated compute budget.These components correspond to sections A.1 through A.4 of the experimental setup.
  • A.1 Models and Inference Configuration: Six publicly released models are evaluated without additional fine-tuning, adapters, or quantization; five use a predefined CROP-AND-ZOOM interface, while Thyme uses Python-generated crop-and-zoom code.The models use officially released checkpoints.
  • A.1 Models and Inference Configuration: The policy comparison holds model weights and decoding parameters fixed while switching between each official tool-use pipeline and a tool-free direct-inference system prompt.Both modes use the same generation configuration, with tool-call handling disabled for direct rollouts.
  • A.1 Models and Inference Configuration: All models are served with vLLM using bf16 weights and native image preprocessing, with one NVIDIA H200 GPU allocated per rollout server.This describes the serving stack used for the experiments.
  • A.2 Benchmarks and Answer Verification: Evaluation covers seven benchmark partitions from four families, with VisualProbe reported as one aggregate column over Easy, Medium, and Hard.Free-form responses are checked first by extracting and normalizing the final answer, then by Qwen3-30B-A3B-Instruct-2507 when exact match fails; either correct result earns a correct score.

A.3 Intervention Implementations … B.3 Qualitative Examples of Regenerated Think Blocks

The appendix specifies leakage-controlled policy, trajectory, and step interventions, then tests corruption choices, intervention-induced trajectory length, and regenerated reasoning. Results show corruption sensitivity varies by scheme, tool-call growth tracks limit-reaching trajectories, and qualitative behavior ranges from retrying to ignoring corrupted evidence.

  • A.3 Intervention Implementations: Policy intervention substitutes the system prompt while keeping decoding, tool engine, and evaluation protocol identical across tool-use and direct modes.
  • A.3 Intervention Implementations: Trajectory intervention corrupts every returned observation online, using RANDOM-CROP, NOISE, or BLANK while preserving each rollout’s shape and tool-call limit.Online corruption prevents later thoughts and actions from inheriting information from clean observations; replaying original thoughts could leak such information.
  • A.3 Intervention Implementations: Step-level VEG averages three independently sampled counterfactual crops and reads option-restricted next-token logits at matched checkpoints before and after observations.The readout appends <answer> to the current thinking prefix rather than running an extra free-form answer rollout.
  • B Additional Trajectory-Level Observation Corruptions: The appendix extends trajectory-level analysis with NOISE and BLANK results, trajectory-length statistics, qualitative regenerated <think> blocks, force-answer ablations, and repeated-seed runs.
  • B.1 V∗Corruption-Choice Ablation: Qwen3-VL shows a markedly smaller V∗ drop under BLANK than RANDOM-CROP, indicating corruption detection but vulnerability to statistically plausible observations.
  • B.2 Trajectory Length under Intervention: Tool-call growth under intervention is highly consistent with the fraction of trajectories reaching the tool-call limit.The comparison uses the mean number of tool calls per item in the intervened condition versus the vanilla baseline on V∗ across six models.
  • B.3 Qualitative Examples of Regenerated Think Blocks: Qualitative rollouts show Mini-o3 retries after an inconsistent RANDOM-CROP, Qwen3-VL-8B continues from the original image after BLANK, and DeepEyes answers despite NOISE.The DeepEyes example commits to A. rubber without acknowledging the corrupted observation.

B.4 Force-Answer Ablation Experiment: Separating Evidence Loss from Truncation · B.5 Results of Repeated Runs · C Additional Analysis of Step-Level Intervention on MCQ Benchmarks

Force-answer ablation shows that trajectory corruption harms performance through both truncated repair loops and genuine visual-evidence loss, with Mini-o3 especially brittle. Repeated random-crop runs preserve large degradation, while the appendix extends analysis of step-level intervention through uncertainty, per-position VEG, and POER comparisons.

  • B.4 Force-Answer Ablation Experiment: Separating Evidence Loss from Truncation: Force-answer reruns preserve corrupted-observation histories while requiring an immediate final answer at the tool-call limit, isolating truncation from evidence loss.The intervention removes failures caused only by reaching the tool-call budget without answering.
  • B.4 Force-Answer Ablation Experiment: Separating Evidence Loss from Truncation: Forced answering recovers much of Qwen3-VL’s original drop, but performance remains below clean tool-use accuracy for every reported model.Qwen3-VL changes from 30.4/38.2 to 73.0/69.6; remaining drops are 18.1 pp for Qwen3-VL-8B, 16.8 pp for Qwen3-VL-4B, and 51.2 pp for Mini-o3.
  • B.4 Force-Answer Ablation Experiment: Separating Evidence Loss from Truncation: The main trajectory-level effect therefore reflects the total impact of corrupting observation feedback, including both evidence removal and induced repair behavior.The force-answer results show that truncation explains only part of the degradation.
  • B.5 Results of Repeated Runs: The repeated-run analysis holds checkpoints, prompts, decoding configurations, benchmark items, and scoring fixed while varying seeds controlling decoding and corrupted-crop sampling.This design characterizes variability across stochastic trajectory-level rollouts.
  • B.5 Results of Repeated Runs: Independent random-crop runs retain substantial degradation despite stochastic variation in corrupted accuracy across Mini-o3 and both Qwen3-VL models.Accuracies were 23.6%, 22.5%, and 24.6% for Mini-o3; 30.4%, 36.1%, and 28.8% for Qwen3-VL-8B; and 38.2%, 46.1%, and 42.9% for Qwen3-VL-4B.
  • C Additional Analysis of Step-Level Intervention on MCQ Benchmarks: The step-level intervention appendix reports repeated-run uncertainty results and per-position VEG details on V∗ to analyze within-trajectory dynamics.These analyses are presented under §§C.1 and C.2.
  • C Additional Analysis of Step-Level Intervention on MCQ Benchmarks: It also compares POER with the raw saturation rate conceptually, while deferring the open-ended extension to Appendix D.The POER comparison is discussed in §C.3.

C.1 Results of Repeated Runs and Uncertainty · C.2 Per-Position VEG Detail · C.3 POER vs. Saturation Rate

Repeated-seed results show stable evidence-use patterns and quantify uncertainty, while per-position VEG analysis and POER distinguish trajectory-level evidence extremes from saturation caused by earlier tool calls.

  • C.1 Results of Repeated Runs and Uncertainty: Four independent seeds were used to measure variability while holding the checkpoint, prompt, decoding, benchmark items, and scoring protocol fixed.Seeds controlled both stochastic decoding and counterfactual-crop sampling.
  • C.1 Results of Repeated Runs and Uncertainty: 85.1%±1.8 pp tool-use accuracy and 0.0629±0.0041 mean VEG were observed for Mini-o3 across four seeds.Values are means ± sample standard deviations.
  • C.1 Results of Repeated Runs and Uncertainty: 90.6%±0.6 pp tool-use accuracy and 0.1944± 0.0183 mean VEG were observed for Qwen3-VL-8B across four seeds.Values are means ± sample standard deviations.
  • C.1 Results of Repeated Runs and Uncertainty: The correct/incorrect VEG split remained stable: both models were positive on correct trajectories and negative on incorrect trajectories.Mini-o3 ranges were 0.078–0.092 versus −0.032–−0.006; Qwen3-VL-8B ranges were 0.237–0.306 versus −0.268–−0.083.
  • C.2 Per-Position VEG Detail: Per-trajectory analysis reports peak V max = maxi VEGi and trough V min =mini VEGi, split by trajectory correctness.The pooled peak column reproduces Table 3’s rightmost column, while trough VEG complements harmful-rate analysis by quantifying worst-case behavior.
  • C.2 Per-Position VEG Detail: Bins with very small n are flagged in the per-position VEG analysis.The appendix identifies this as a limitation of the fine-grained analysis.
  • C.3 POER vs. Saturation Rate: POER measures continued tool calling after confidence crosses the saturation threshold, unlike saturation rate Pr[gi−1 >τsat].It conditions jointly on g0 ≤τsat and k ≥2, restricting analysis to trajectories requiring tool use and attributing saturation to earlier policy calls.

D Step-Level Intervention on Open-Ended Benchmarks … E.3 Behavioral Validity for Group Labels

The appendix extends step-level visual-evidence auditing to open-ended VisualProbe and validates the diagnostic through threshold sensitivity, policy-ATE decomposition, and independent behavioral signatures. Open-ended results preserve the qualitative MCQ patterns but require separate calibration, while behavioral audits support the Mode 1 and Mode 2 labels.

  • D Step-Level Intervention on Open-Ended Benchmarks: Open-ended VEG extends the step-level intervention beyond MCQ by measuring whether real returned crops increase length-normalized support for the gold answer over counterfactual crops.The extension is supplementary because open-ended VEG is unbounded nats per token and is not directly comparable to bounded MCQ VEG.
  • D Step-Level Intervention on Open-Ended Benchmarks: VisualProbe results reproduce the MCQ directional pattern: DeepEyes shows weak observation-mediated influence, Mini-o3 benefits mainly on correct trajectories, and Qwen3-VL-8B has the largest correct-versus-incorrect separation.Scores are reported as median (IQR) in nats per gold token because the raw open-ended measure has heavy tails.
  • D Step-Level Intervention on Open-Ended Benchmarks: Open-ended VEG does not support transferring the MCQ saturation threshold, near-zero cutoff, or four-group classifier without separate calibration.The score is unbounded, has wide IQRs, and is not a margin against the strongest competing free-form answer.
  • E Diagnostic Classification: Supporting Results: The appendix supplies supporting evidence for the trajectory-level diagnostic through threshold sweeps, group-wise policy-ATE decomposition, behavioral audits, and validation on HR-Bench-4K.These components are organized across E.1–E.4.
  • E.1 Diagnostic Threshold Sensitivity: Across threshold sweeps, DeepEyes remains Mode 1-dominant without Mode 2 items, Qwen3-VL models retain Calibrated tails, and Mini-o3 remains the most Calibrated-heavy model.The Calibrated contribution remains positive and accounts for most of the positive policy-level ATE for Mini-o3, Qwen3-VL-8B, and Qwen3-VL-4B throughout both sweeps.
  • E.1 Diagnostic Threshold Sensitivity: +2.0 pp is the Calibrated subset contribution for Mini-o3 at the boundary cutoff τsat = 0.80.This lenient saturation criterion reclassifies some trajectories that are useful under stricter high-confidence cutoffs.
  • E.2 Group Decomposition of the Policy-Level ATE: Policy-level ATE is decomposed by group as the fraction of each group multiplied by its mean accuracy gap between tool-use and direct inference, with unrounded contributions matching the total.The groups are No-call, Mode 1, Calibrated, and Mode 2.
  • E.3 Behavioral Validity for Group Labels: Mode 1 trajectories commit to their final answer before calling in ∼58% of cases versus ∼28% for Calibrated trajectories, supporting their interpretation as performative calls.This audit covers DeepEyes, Qwen3-VL-8B, and Mini-o3 on V∗.

E.4 Cross-Benchmark Validation on HR-Bench-4K · F Declaration of LLM usage

On HR-Bench-4K, the VEG protocol and diagnostic reproduce the same model-specific signatures and support the same Calibrated-subset conclusion beyond V∗. The authors report using LLMs for prose polishing and Cursor for implementation, while retaining authorship of the scientific work.

  • E.4 Cross-Benchmark Validation on HR-Bench-4K: The study applies the same VEG protocol and deterministic diagnostic rule (τsat =0.95, ϵ=0.01) to 800 HR-Bench-4K items across DeepEyes, Mini-o3, and Qwen3-VL-8B.This tests whether the four-group diagnostic and ATE decomposition generalize beyond V∗.
  • E.4 Cross-Benchmark Validation on HR-Bench-4K: DeepEyes remains largely inactive: 86.9% of calls are near-zero, and its per-trajectory best call is near zero.These results reproduce the inactive-use signature on HR-Bench-4K.
  • E.4 Cross-Benchmark Validation on HR-Bench-4K: Mini-o3 remains information-driven but diluted, with a correct-versus-incorrect gap of +0.115 versus −0.023 and a positive best call of +0.171.The positive best call coexists with weaker aggregate information use on incorrect trajectories.
  • E.4 Cross-Benchmark Validation on HR-Bench-4K: Qwen3-VL-8B remains saturation-dependent: 58.3% of calls are saturated with near-zero VEG, while non-saturated calls remain positive.This reproduces the saturation-dependent signature on HR-Bench-4K.
  • E.4 Cross-Benchmark Validation on HR-Bench-4K: The Calibrated subset is positive for all three models and is the largest contributor for Mini-o3 and Qwen3-VL-8B in the group distribution and ATE decomposition.The same high-level pattern extends beyond V∗.
  • E.4 Cross-Benchmark Validation on HR-Bench-4K: For DeepEyes, the overall negative ATE is driven by Mode 1 and No-call, whose negative contributions outweigh the small positive Calibrated term.Thus, the grouping is not a V∗-only artifact.
  • F Declaration of LLM usage: The authors used LLMs only to polish prose after completing the draft and used Cursor to implement their original design and ideas.They state that the scientific contributions, technical methods, ideas, and core results are entirely their original work.
Loading 2608.06270v1…