PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?

Zeyu Wang, Xinming Xu

arXiv:2608.22916v1cs.CL

TL;DR

VLMs can encode spatial information without using it in answers, leaving unclear when and where that information becomes causally effective. The paper applies direction patching across layers, positions, prompt formats, datasets, and ten VLMs, finding conditional transport that emerges at mid-to-deep layers, is often suppressed by text CoT, and remains open under visual grounding prompts.

  • Problem

    VLMs may encode spatial information in hidden states yet fail to use it when answering, and the local conditions governing causal reach to answer logits remain unclear.

  • Method

    The paper uses direction patching to intervene on class-conditioned spatial-ID directions across layers, token positions, and prompt formats.

  • Results

    Across ten VLMs, spatial transport emerges at L12–L32; text CoT often suppresses immediate object-word argmax-level transport, while visual grounding prompts keep it open.

  • Takeaways & Limitations

    The results characterize the encoding-grounding gap as conditional transport governed by prompt format, layer depth, token position, and route to answer logits.

  • Takeaways & Limitations

    The claims concern causal influence along the tested linear subspace and do not cover transport through nonlinear pathways or alternative directions; head localization is proof-of-concept.

Abstract

from arXiv · show

Vision-language models are known to encode spatial information in their hidden states, yet often fail to use it when answering. However, it remains unclear when and where this encoded information reaches the answer. We address this with direction patching, a class-conditioned causal intervention applied across layers, token positions, and prompt formats. Using spatial-ID directions constructed following prior encoding evidence, we find that causal influence on answer logits emerges only at mid-to-deep depths. Text chain-of-thought suppresses immediate object-word argmax-level transport in most models, while visually grounded prompts keep it open. Positive target-logit gain can remain below the argmax threshold, and transport can re-emerge at the final prefix token or at the answer step in deeper layers. Across the ten VLMs we study, these local effects form descriptive transport patterns. Complementary experiments characterize how these patterns shift across datasets, attributes, and encoding amplitudes. Together, these results reframe the encoding-grounding gap as a problem of conditional transport in VLMs.

1 Introduction

The paper studies why encoded spatial information in VLM hidden states often fails to influence answers, tracing its causal transport across layers, positions, and prompt formats. Direction patching reveals conditional transport: text CoT commonly suppresses immediate object-word effects, while visual grounding prompts preserve them.

  • Motivation: The paper reframes the encoding-utilization dissociation as an unresolved question of when and where encoded visual information causally reaches answer logits.It moves beyond static representation analysis to trace transport conditions inside prompts.
  • Method: Direction patching manipulates class-conditioned hidden-state directions at controlled layers and token positions to isolate their causal effects on answer logits.The study builds on previously identified spatial-ID directions and maps their transport across layer, position, and prompt format.
  • Results: Across ten VLMs, causal spatial transport emerges late, activating at L12–L32 despite spatial-ID directions being constructed from early-layer representation evidence.This is one of three reported local regularities.
  • Results: Text CoT suppresses immediate object-word argmax-level transport in most models, whereas visual grounding prompts keep the pathway open.The broader result is conditional on prompt format, token position, and layer depth.
  • Results: At the model level, local transport effects form descriptive groupings, while behavioral steering and a proof-of-concept head knockout provide additional architecture-dependent evidence.The experiments also identify visual grounding prompts as a prompt-only intervention that keeps the object-word pathway open.

2 Experiment

The experiments evaluate spatial and color reasoning across multiple datasets, models, layers, token positions, and prompt formats. They use crossfit class-conditioned directions and compare answer effects against same-norm random-direction baselines.

  • Tasks: The study evaluates three four-way MCQ tasks sharing a common pipeline but targeting different visual attributes.RefCOCO and GQA provide spatial tasks, while GQA color changes the target attribute and patched direction.
  • Models: The evaluation covers ten open-weight VLMs from five families, including within-family scale comparisons for Qwen, InternVL3, and Gemma-3.All models expose residual-stream activations required for direction patching.
  • Intervention: Class-conditioned centroids are computed at each layer from object-word hidden states using five-fold crossfit, then residual-stream activations are shifted at selected positions.The model completes a forward pass, and next-token logits are read over {A, B, C, D}.
  • Metrics: The primary diagnostic compares target flip-rate changes against same-norm random-direction baselines, alongside continuous target-letter logit gain.A zero argmax effect does not imply zero target-logit gain, and the diagnostic is limited to the tested linear subspace.
  • Conditions: Interventions target obj_word and prefix_last across seven suffix formats, including text-reasoning and visually grounded prompts.The prefix_last experiments reuse the object-word-derived spatial direction rather than recomputing a position-specific centroid.

3 Local Dynamics of Conditional Transport

Spatial directions become causally effective only under specific layer, prompt, and token-position conditions: transport emerges after early encoding, text CoT often gates the object-word route, visual grounding preserves it, and deeper layers can reopen transport at later positions.

  • 3.1 Spatial Encoding Does Not Immediately Become Transport: Transport activates at mid-to-deep layers despite spatial-ID directions being constructed from early-layer representation evidence.Across models, encoding availability is concentrated around L4–L8, whereas transport appears around L12–L32.
  • 3.1 Spatial Encoding Does Not Immediately Become Transport: Shallow layers produce near-zero object-word ∆argmax, while transport peaks later, including Janus L12 +0.325 and Qwen-32B L32 +0.089.The larger model reaches its peak at a higher absolute layer index, with comparable relative depth.
  • 3.1 Spatial Encoding Does Not Immediately Become Transport: Wrong-axis and non-object-position controls remain near zero, supporting direction- and position-specific causal effects.Their ranges are [−0.018, +0.031] and [−0.017, +0.006], respectively.
  • 3.2 CoT Suppresses Immediate Object-Word Transport: Text CoT suppresses object-word argmax-level transport in eight of ten models, with peaks at or below 0.006.Janus falls from +0.325 under answer-only to below the descriptive near-zero threshold; Qwen-7B and InternVL2.5 are exceptions.
  • 3.2 CoT Suppresses Immediate Object-Word Transport: The CoT block can affect argmax flips without erasing the underlying signal: InternVL3-8B has ∆argmax 0 but target-letter logit gain +0.421.Increasing intervention amplitude does not systematically restore the suppressed models’ argmax-level transport.
  • 3.3 Visual Grounding Bypasses the Gate: Visual grounding preserves object-word transport across all ten models, including models suppressed by text CoT.Visual CoT exceeds answer-only for Qwen-32B, InternVL2.5, InternVL3-14B, and Janus; Janus reaches +0.403 versus +0.069.
  • 3.4 Transport Pathways Are Position-Dependent: Transport is position-dependent: object-word effects can decline while deeper prefix-last effects re-emerge.InternVL3-8B shows object-word L12 +0.112 and prefix-last L20 +0.455, with CoT gated only at object-word.
  • 3.4 Transport Pathways Are Position-Dependent: Models differ in pathway organization: Janus concentrates transport at object-word, while Gemma-3 shows weak, dataset-dependent prefix-last effects.Janus has object-word +0.315 at L12 and negative prefix-last ∆argmax.

4 Model-Level Patterns and Boundary Conditions

Across ten VLMs, local transport effects organize into descriptive model-level patterns that vary with prompt format, architecture, dataset, attribute, and encoding amplitude. Behavioral and head-level probes further show that logit transport can extend beyond immediate argmax effects, while its validation and interpretation remain architecture- and setting-dependent.

  • 4.1 Transport Groupings Across Models: Four descriptive transport patterns organize the ten models, with reduced/suppressed relocating and suppressed non-relocating each covering 4/10 models.The remaining models are Qwen-7B, which is open, and Qwen-32B, which is prompt-selective.
  • 4.1 Transport Groupings Across Models: Qwen-7B remains open to CoT transport, whereas Qwen-32B shows prompt-selective prefix_last transport: +0.431 with final_tag versus +0.019 answer-only at L55.
  • 4.1 Transport Groupings Across Models: Janus combines strong visual_cot object-word transport, rising from +0.069 to +0.403, with no generate-mode accuracy change under steering.This provides a boundary case where a substantial logit effect does not align with behavioral change.
  • 4.2 Behavioral Consequences and Head-Level Probe: Under answer-only, incorrect-quadrant steering reduces accuracy by 8–21pp in nine of ten models, while intact CoT baselines still show 13–16pp drops under object-word intervention.The intervention occurs before autoregressive generation, so later hidden states and parsed answers can change without an immediate answer-option argmax flip.
  • 4.2 Behavioral Consequences and Head-Level Probe: In InternVL3-8B, knocking out L27.H21 recovers +0.063 CoT ∆argmax, while the comparable LLaVA-OV scan reaches only +0.013, indicating architecture-dependent head concentration.H21 has no answer-only effect (+0.007), supporting a CoT-specific gating role in this scan.
  • 4.3 Generality and Boundary Conditions: Transport-map structure transfers across datasets in 9/10 suppression cases and 8/10 relocation cases, while color transport is gated in 10/10 models and peaks ∼4–8 layers earlier than spatial transport in 9/10.Amplitude has no universal cross-architecture threshold, although it can flip gating within Qwen-7B.

5 Related Work

Prior work largely probes what VLMs encode or correlates attention with processing, while this paper localizes when encoded spatial information causally reaches answers under different prompts.

  • Static probes establish that VLMs can encode spatial properties while failing to use them at output.
  • Attention inspection maps candidate heads and grounding circuits, but attention overlap remains correlational.
  • Text CoT can degrade spatial accuracy, yet prior work does not localize where the pathway is blocked.
  • This analysis complements decoding-time grounding methods by tracing upstream spatial transport across layers and token positions.

6 Conclusion

The paper shows that spatial transport is conditional rather than uniformly available: layer depth, prompt type, and token position determine whether encoded information influences answer logits.

  • Direction patching traces when encoded spatial information becomes causally available for VLM answers.
  • Spatial-ID directions can precede answer-logit effects, while text CoT often suppresses immediate object-word transport and visual grounding keeps it open.
  • Across models, local effects form descriptive transport groupings supported by behavioral steering and architecture-dependent head knockout evidence.
  • Immediate argmax suppression does not by itself establish trajectorywide erasure because answer-step and stepwise controls reveal later routes.

Limitations

The study’s conclusions are bounded by its linear intervention subspace, selected tasks and models, single-pass validation, manually designed prompts, and proof-of-concept circuit analysis.

  • Direction patching measures causal influence only along the tested class-conditioned linear subspace, not nonlinear or alternative-direction transport.
  • The evaluation covers spatial position and color in a four-way forced-choice format, excluding freeform descriptions, fine-grained localization, and other attributes.
  • All ten models are open-weight and range from 4B to 32B parameters, so larger or closed-source architectures may differ.
  • Generate-mode validation uses 128-token single-object interventions, and longer generation or multiturn interaction may produce different accuracy patterns.
  • Visual grounding prompts are manually designed, leaving systematic wording optimization outside the study’s scope.
  • Head localization is proof-of-concept: one InternVL3-8B gating head is identified, while upstream, downstream, and cross-family mechanisms remain open.
  • The prompt manipulation holds the question stem fixed and varies only the appended instruction suffix.
  • The prompt ladder spans reasoning length and visual grounding formats, with underlying peak and threshold values reported for the tested models.

C Controls

Controls show that observed effects are direction- and position-specific, while additional analyses distinguish argmax thresholds, dose responses, and prefix-token transport patterns.

  • Direction and position controls: Matched controls address whether object-word effects arise from perturbation magnitude or token sensitivity rather than spatial content.
  • Direction and position controls: Both wrong-axis and non-object patches stay within ±0.03 across ten models, far below Janus’s target +0.325 effect.
  • Direction and position controls: The target effect requires both the correct spatial direction and the correct object-word position, including on COCO-Spatial.
  • Bootstrap uncertainty: Bootstrap intervals support distinguishing exact-zero from borderline near-zero argmax effects without making the descriptive threshold a universal significance test.
  • Argmax versus logit gain: Even with IV3-8B CoT ∆argmax at zero, target logit gain is +0.421 with 95% CI [+.341, +.501], indicating sub-threshold directional influence.
  • Amplitude sweeps: Across α ∈{1, 2, 5, 10}, answer-only ∆argmax grows while suppressed models remain near zero at obj_word.
  • Amplitude sweeps: At prefix_last, Qwen-7B CoT peaks at α=5 (+.356) before falling to α=10 (+.069), suggesting earlier saturation than obj_word.
  • Prefix-last prompt ladder: Prefix_last transport varies from prompt-stable to prompt-selective or near-zero, with Qwen-32B showing +0.431 versus +0.019 across formats.

G Generate-Mode Accuracy

Generate-mode interventions show that spatial transport can affect parsed answers even when immediate object-word argmax transport is absent, but effects depend strongly on prompt, model, and position. Cross-dataset tests preserve these model-specific transport patterns while exposing architecture-linked exceptions.

  • Generate-mode accuracy: Steering toward incorrect quadrants reduced answer-only accuracy by 8–21 percentage points in nine of ten models.Examples include Qwen-7B dropping from 74.2% to 53.6% and IV3-8B from 77.5% to 56.9%.
  • Prompt dependence: Under text CoT, near-chance-baseline models showed at most 2.2-point further drops, while intact-baseline models still dropped 13–16 points.These accuracy changes occurred despite near-zero immediate object-word ∆argmax in the latter group.
  • Delayed transport: Object-word perturbations can affect later autoregressive states and final parsed answers without changing the first answer-option argmax.Thus, immediate argmax transport does not exhaust the possible answer pathway.
  • Model exceptions: Visual_cot steering produced sizable accuracy drops for many models, but IV2.5, Gemma-3, and Janus showed weaker or absent changes.Janus had the strongest visual_cot ∆argmax (+0.403) yet generated-answer accuracy changed by at most 3 points under any prompt.
  • Cross-dataset validation: GQA preserved suppression, peak-layer stability within ±8 layers, and relocation for suppressed–relocating models.Janus remained non-relocating with negative prefix_last transport, while Gemma-3-27B showed a weak GQA prefix_last effect.

I GQA Color Sweep

The attribute sweep shows that transport behavior depends on both the visual attribute and encoding amplitude. Color transport is gated by CoT in every model, peaks earlier than spatial transport, and can fall below the amplitude needed to sustain transport.

  • GQA Color Sweep: CoT gated color transport in 10/10 models, compared with 8/10 for spatial transport.Color peaks occurred four to eight layers earlier than corresponding spatial peaks.
  • Amplitude boundary: Qwen-7B’s color answer-only peak was only +0.082, and CoT color ∆argmax collapsed to 0.000.The architecture, pipeline, and intervention were unchanged; only the attribute encoding amplitude differed.
  • Visual grounding: Both visual prompts bypassed CoT gating at object_word in every gated model.For IV3-8B, Vis-C reached +.164 and Vis-D +.144 while CoT remained .000.
  • Visual grounding: Visual grounding sometimes opened routes unavailable under answer-only, including Janus object-word values roughly six times its answer-only peak.Janus reached +.403 with Vis-C and +.397 with Vis-D versus an answer-only peak of +.069.
  • Prompt-class robustness: Vis-C and Vis-D agreed closely across models, with Pearson correlation r > 0.99 at object_word.The largest discrepancy was Gem-12B, at +.064 versus +.103.
  • Axis control: Horizontal-axis centroids did not shift vertical-axis answers, and vertical centroids did not shift horizontal-axis answers.The result supports direction-specific rather than generic centroid sensitivity.

O Qwen-32B Scaling Sanity

The Qwen scaling sanity check supports deeper-layer emergence in the larger model while revealing a scale-dependent prompt pattern. Qwen-32B remains CoT-gated and shifts from Qwen-7B’s open pattern to a prompt-selective pattern.

  • Layer scaling: Qwen-32B answer-only ∆argmax increased monotonically from +.019 at L16 to +.089 at L32, while CoT remained 0.000.A larger n=168 rerun confirmed +.085 with a 95% bootstrap interval of [.060, .111].
  • Layer scaling: The deeper Qwen-32B layer pattern mirrors Qwen-7B’s late emergence at a proportionally deeper layer index.Qwen-32B has 64 layers compared with Qwen-7B’s 28.
  • Prompt pattern: Qwen-7B has an open object-word route under both prompts, whereas Qwen-32B has null CoT transport even at its deepest answer-only peak.The family therefore changes from open to prompt-selective between 7B and 32B.
  • Prompt pattern: The cross-scale change makes family an insufficient predictor of transport pattern within the Qwen family.This is presented as a within-family counter-example.

Q Attention Head Knockout Details

Head knockout localizes CoT gating to one InternVL3-8B head but finds no similarly concentrated gate in LLaVA. The experiments therefore distinguish gate existence from architecture-dependent gate localization.

  • InternVL3-8B: InternVL3-8B head L27.H21 exceeded the +0.03 threshold, recovering +0.062 at n=50 and +0.063 in an n=100 confirmation.All other 124 scanned head–layer cells were at or below threshold.
  • InternVL3-8B: The same InternVL3-8B head produced only +0.007 under answer-only, arguing against a generic always-helpful effect.The recovery is specific to the CoT-gated condition in this probe.
  • LLaVA: LLaVA’s best candidate, L22.H25 at +0.040, shrank to +0.013 in the n=100 confirmation and remained +0.003 under answer-only.No scanned LLaVA head exceeded the +0.03 threshold robustly.
  • Scope: Qwen-7B was not tested with head knockout because its object-word pathway is open and has no gate to localize.A full upstream–downstream circuit account remains open.
Loading 2608.22916v1…