Source-linked AI summary
Readout is not Recovery: Dissociating Coordinate Emission from Visual-Corruption Repair in Vision-Language Models
Drandreb Earl Juanico
TL;DR
VLMs can emit parseable localization boxes, but whether dimensions supporting coordinate emission also support repair after visual damage is unresolved. The paper compares clean coordinate-token rankings with corruption-derived rankings using depth triage and endpoint-replacement recovery. Their depth profiles differ, and endpoint readout evidence alone is not sufficient to establish recovery from damaged visual evidence.
Problem
Parseable localization outputs make grounding measurable, but it remains unclear whether dimensions that support coordinate emission also support repair when visual evidence is damaged.
Source: §1 · pp. 1–2Method
The study compares clean coordinate-token readout rankings with corruption-derived rankings using clean-input depth flooring and clean endpoint replacement after object masking or partial occlusion.
Source: §1 · p. 2Results
Coordinate-token rankings are load-bearing only at later Qwen3-VL layers, while corruption-derived rankings harm layers 16–24 but help near layer 35 and the final site.
Source: §1 · p. 3, §6 · p. 14Takeaways & Limitations
Endpoint rankings are useful circuit priors for localization, but recovery from damaged visual evidence requires separate corruption-specific causal tests.
Source: §6 · p. 14The synthetic masking and partial-occlusion probes support mechanistic conclusions under occlusion-like damage, not deployment claims about natural occluders, clutter, motion, or multi-instance scenes.
Source: §5 · p. 13
Abstract
from arXiv · showhide
VLM bounding-box localization is both language generation and spatial commitment. Parseable fields such as bbox_2d make localization easy to score, but dimensions that emit coordinate tokens need not repair localization after visual evidence is damaged. We study this readout/recovery separation in Qwen3-VL-4B-Instruct on single-object COCO grounding. We compare clean coordinate-token readout rankings with corruption-derived repair rankings, using object-mask endpoint replacement for recovery and clean-input flooring for depth localization. In Qwen3-VL, coordinate-token rankings are inert through layer 24, load-bearing from layers 32-35, and peak at layer 34; corruption-derived rankings harm layers 16-24 but become beneficial near layer 35/final. A Kimi-VL-A3B diagnostic shows a matching output-proximal transition despite a different box format. Object-mask recovery separates rank budgets: $k=250$ shows necessity, $k=500$ shows Top-$k$ restoration above random, and $k=d/2$ is largely capacity-driven. Partial-occlusion sweeps reveal that high-overlap coordinate-token sets can hurt at $k=1000$ and help mainly at half-width, while population corruption-derived sets provide no reliable fixed repair set. Edge-attribution patching shows coordinate-token paths are high precision but low recall for detection recovery, and RMSNorm quasi-layer controls do not close the endpoint-repair gap. Endpoint coordinate triage is therefore a useful circuit prior, but occlusion recovery requires a separate benchmark.
1 Introduction
The paper asks whether dimensions that support coordinate emission also preserve localization when visual evidence is damaged. It separates clean readout rankings from corruption-derived repair rankings and finds that their depth profiles and recovery behavior differ.
1 Introduction
- Structured localization makes grounding measurable, but emitting a parseable box does not establish that the model preserves the visual evidence needed for correctness.Generated boxes can be parsed and scored against annotated objects using COCO-style localization metrics.Source: §1 · p. 1
- Three rankings distinguish output-interface leverage from repair leverage: bbox_2d readout, numeric coordinate-token readout, and corruption-derived alignment with clean-to-object-mask repair displacements.The first two use clean readout behavior; the third aggregates repair displacements across the ranking population.Source: §1 · p. 2
- Object masking blanks the queried object region, whereas partial occlusion leaves some object evidence visible and serves as a softer, occlusion-like probe.Neither synthetic corruption is presented as a complete naturalistic occlusion benchmark.Source: §1 · p. 2
- Coordinate-token rankings are inert through layer 24, load-bearing from layers 32–35, and null at the final readout site, while corruption-derived rankings harm layers 16–24 but help near layer 35 and the final site.Object-mask recovery also separates rank budgets: coordinate-token Top-k sets move from necessity at k = 250 to restoration above random at k = 500, while half-width recovery is largely capacity-driven.Source: §1 · pp. 2–3
- The study frames grounding interpretability as a distinction between coordinate-emission geometry and visual-corruption recovery geometry.It defines object-mask repair through clean endpoint replacement and reserves flooring for clean-input depth localization.Source: §1 · p. 3
2 Related Work
Prior work establishes structured localization outputs and endpoint-based circuit tracing as useful foundations. This paper adds a separate test of whether endpoint rankings transfer to repair after visual evidence is corrupted.
Structured localization in VLMs
Endpoint tracing and causal intervention
- Endpoint-based causal tracing identifies components implicated in behavior, while this study adds a transfer test from clean localization readout to visual-corruption repair.Related approaches include causal mediation and tracing, path patching and IOI, and ACDC.Source: §2 · p. 4
Faithfulness, corruption, and normalization
- Attribution can identify output-facing sensitivity without proving implementation, and normalization can complicate dimension-level interventions, motivating a quasi-layer RMSNorm control.Prior work discusses attribution faithfulness, visual corruption and occlusion [13], and RMSNorm and LayerNorm [2,21].Source: §2 · p. 4
3 Method
The method compares clean readout rankings with corruption-derived rankings using separate clean-input depth localization and corrupted-input endpoint-replacement recovery tests. It evaluates these interventions on single-object COCO grounding, with a targeted Kimi-VL depth-triage check.
Model, data, and endpoint
- Qwen3-VL-4B-Instruct is evaluated on single-object COCO val2017 prompts, with localization scored by R50, recall at IoU ≥0.50, and unparseable outputs assigned zero recall.The primary endpoint is the final RMSNorm output before the LM head; interventions run only during localization-token decoding.Source: §3 · pp. 4–5
Cross-model Kimi-VL diagnostic
- A targeted Kimi-VL-A3B diagnostic ranks dimensions from generated coordinate tokens and floors the Top-1024 of its 2048-wide hidden state across decoder depths and the final norm output.It is a cross-model readout-triage check, not a second full recovery benchmark.Source: §3 · p. 5
Readout directions and rankings
- Clean readout rankings score hidden dimensions by their contribution to target-token readout, aggregated separately at bbox_2d field-token and numeric coordinate-token positions.Corruption-derived rankings instead aggregate dimensions aligned with clean-to-object-mask endpoint displacements across ranking samples.Source: §3 · p. 5
Clean-input depth triage
- Clean-input depth triage floors selected dimensions with an elementwise clamp at θ = −0.5 to locate where a ranking is load-bearing, not to measure recovery.Values above the floor remain unchanged, and the threshold is not claimed to be layer-calibrated.Source: §3 · p. 6
Endpoint recovery under visual corruptions
- Corruption recovery masks the full object or applies partial occlusion, then replaces selected corrupted endpoint dimensions with corresponding clean-trace values.Object masking is the hard repair benchmark; partial occlusion is a softer probe that leaves some object evidence visible.Source: §3 · p. 6, Figure 1 · p. 7
Evaluation suite
- The evaluation suite combines depth flooring, rank-budget and local-set endpoint recovery, partial-occlusion edge-attribution patching, and a quasi-layer RMSNorm control.Local-set comparisons include fixed, split-local, and sample-specific rankings, while edge rankings are validated by zero-ablation.Source: §3 · p. 7
4 Results
Clean coordinate-token rankings and corruption-derived rankings have distinct, non-monotonic depth profiles. Endpoint recovery also depends on rank budget and corruption setting, so clean readout rankings do not provide a universal repair ranking.
4 Results
- Qwen coordinate-token Top-1280 dimensions shift from complement-load-bearing at layers 21–24 to top-load-bearing at layers 32–35, peaking in specificity at layer 34.The population corruption-derived ranking is harmful at layers 16–24 and dual-criterion load-bearing at layer 35.Source: Table 1 · p. 8, §4 · pp. 8–10, Table 3 · p. 9
- Kimi-VL-A3B shows a late coordinate-token transition, with a step change at layer 26 and a stronger final-site effect despite using plain normalized boxes.The diagnostic uses Top-1024, half of Kimi's 2048-dimensional hidden width.Source: Table 2 · p. 8, §4 · p. 9
- At k = 500, coordinate-token Top-k restoration beats random (10.60% versus 8.77%, z = +2.35), while k = 250 shows necessity and k = 1280 = d/2 is capacity-dominated.Population corruption-derived rankings show exclusion and inclusion effects at k = 500, with modes converging near 69–70% at half-width.Source: Table 4 · p. 10, §4 · p. 10
- Clean coordinate-token rankings transfer at k = 500, fall below the high-variance random baseline at k = 1000, and converge with other non-oracle sets near half-width.The local-set comparison includes fixed, split-local, sample-local, sample-specific corruption, and bottom-ranked sets.Source: Table 5 · p. 10, §4 · p. 10
- Under partial occlusion, fixed and split-local coordinate-token sets with near-perfect overlap are below random at k = 1000, while several readout sets turn positive at k = 1280.Population- and sample-specific corruption-derived sets do not provide a reliable fixed repair set across the reported budgets.Source: Table 6 · p. 11, §4 · pp. 11–12
- With up to 100 ablated edges, coordinate-token Top-k paths are slightly more behaviorally concentrated, but at 200 edges random and complement sets damage detection more.Coordinate-token paths change J most, while broader detection-critical partial-occlusion behavior also uses other pathways.Source: §4 · p. 12
- Object-mask recovery raises parseability from a 62% corrupted baseline to about 97% for any k = 500 restoration, while partial-occlusion coordinate-token deficits persist after conditioning on parseable outputs.The reported format collapse therefore does not account for the ranked-set-specific recovery pattern.Source: §4 · p. 12
- Making RMSNorm explicit leaves repair rankings largely unchanged, and transporting endpoint attribution through its Jacobian changes endpoint-repair overlap only slightly.The dissociation remains visible in both post-norm and pre-norm spaces.Source: §4 · p. 12, Table 7 · p. 13
5 Limitations and Future Work
The study is endpoint-centered and uses synthetic occlusion-like damage as a probe, not a complete robustness benchmark. Future work proposes gated, conservative endpoint-based repair using pseudo-clean completion and stability filtering.
5 Limitations and Future Work
- The interventions test residual-stream dimensions at the endpoint, not the full upstream visual pathway, and the targeted depth grid does not exhaustively sample layers.Vision-encoder representations, cross-modal attention, and intermediate MLP computations may contain repair-relevant structure.Source: §5 · p. 13
- Object masking is less naturalistic than many real occlusions, while partial occlusion remains synthetic and fixed across intervention conditions.Conclusions are mechanistic evidence under occlusion-like damage, not deployment claims about natural occluders, clutter, motion, or multi-instance scenes.Source: §5 · p. 13
- Future work proposes gated, conservative repair using visible-region-conditioned pseudo-clean completion and stability filtering, tested without deployment-time ground-truth boxes.The proposed pipeline combines occlusion triggers, completion candidates, and selective patching or reweighting of recovery-relevant dimensions.Source: §5 · p. 13
6 Discussion and Conclusion
The paper distinguishes clean coordinate emission from recovery when visual evidence is damaged. Its endpoint rankings are useful circuit priors, but recovery requires separate corruption-specific tests.
6 Discussion and Conclusion
- Coordinate-token rankings become load-bearing after Qwen3-VL’s layer-24-to-layer-32 transition, peak at layer 34, and show a matching output-proximal transition in Kimi-VL.Population corruption-derived rankings instead harm layers 16–24 and become beneficial late; object-mask half-width restoration is largely capacity-driven, while lower-budget repair evidence is strongest for corruption-derived indices.Source: §6 · p. 14
- Under partial occlusion, coordinate-token fixed and split-local sets are negative at k = 1000 but nonnegative at half-width, while population corruption-derived sets remain null or sub-random.Readout-targeted edges are precise at small budgets but do not cover the broader detection-critical pathways; explicit final-normalization controls do not close the endpoint-repair gap.Source: §6 · p. 14
- Endpoint rankings are principled localization circuit priors, not sufficient evidence that the visual grounding pathway has been identified.Recovery from damaged visual evidence requires corruption-specific causal tests rather than endpoint-readout evidence alone.Source: §6 · p. 14
A.1 Notation summary
The notation summary defines the indices and sets used for dimensions, interventions, and visualization. The supplementary material includes code for reproducing the intervention pipeline and its outputs.
A.1 Notation summary
- The notation uses i for sample, t for decode-token position, q for vocabulary token, j and p for hidden dimensions, and ℓ for decoder layer.The hidden width is d = 2560; r denotes a dimension ranking, I an intervention set, S a visualization set, and c an intervention condition.Source: §A.1 · p. 16
- The anonymized supplementary material includes code for reproducing intervention pipelines, table aggregation, and figure generation.Source: §A.1 · p. 16
A.2 Parseability convention
The paper separates whether a localization response can be parsed from whether its box is geometrically accurate. Unparseable intervention outputs are retained in evaluation rather than filtered out.
A.2 Parseability convention
- A response is parseable when it contains bbox_2d and numeric coordinates that can be converted into a valid four-coordinate box after consistent post-processing.Post-processing includes coordinate conversion, ordering, and clipping to image bounds where applicable.Source: §A.2 · p. 16
- Intervention evaluation does not constrain decoding or filter outputs after generation, so omitted fields or invalid boxes count as unparseable.Clean readout-ranking examples, by contrast, must have parseable localization responses.Source: §A.2 · p. 16
- Unparseable outputs count as zero localization recall, while R50 | parse evaluates only parseable generations; Table 8 separates format failure from spatial error.The table covers clean-input depth triage, cross-model Kimi-VL depth triage, object-mask endpoint recovery, and partial-occlusion local-set recovery.Source: §A.2 · p. 16, Table 8 · p. 17
- For object-mask recovery, any k = 500 restoration largely repairs parseability, whereas partial-occlusion localization error dominates when the object remains partly visible.Table 8 is an effect-size diagnostic rather than a replacement for the primary R50 metric.Source: §A.2 · pp. 16–17
A.3 Sites, token positions, and hook convention
The final readout site is after final RMSNorm and before the LM head; decoder-layer sites are decoder-block outputs. Hooks operate only during generated localization-token steps, not image/prompt prefill.
A.3 Sites, token positions, and hook convention
- The final readout site is the output of final RMSNorm immediately before the LM head, while decoder-layer sites are outputs of their corresponding decoder blocks.Source: §A.3 · p. 17
- Hooks fire only during autoregressive generation steps with sequence length one and do not alter the image/prompt prefill pass.Source: §A.3 · p. 17
- Target positions are generated localization tokens: bbox_2d, following coordinate-value tokens, or both, depending on the ranking.The token index refers to generated decode positions, not image-token positions.Source: §A.3 · p. 17
A.4 Kimi-VL coordinate-token depth-triage diagnostic
The Kimi-VL diagnostic applies coordinate-token depth triage to bracket-form boxes rather than Qwen3-VL’s bbox_2d JSON convention. It evaluates a half-width dimension set across proportional decoder depths and the final norm output.
A.4 Kimi-VL coordinate-token depth-triage diagnostic
- Kimi-VL generations use plain normalized boxes in brackets, [x1, y1, x2, y2], rather than Qwen3-VL’s bbox_2d JSON convention.Source: §A.4 · p. 17
- The diagnostic uses coordinate-token triage, constructing a token-specific contrast direction from the untied language-model head at each bounding-box decode step.The reported dimension score weights the change in hidden state by the contrast direction, with θ = -0.5.Source: §A.4 · p. 18
- The 2048-dimension ranking uses k = 1024 across proportional sites in Kimi’s 27-layer decoder and the final norm output.The reported diagnostic is coordinate-token depth triage only; a Kimi population corruption-derived recovery ranking is outside its scope.Source: §A.4 · p. 18
A.5 Contrast readout directions
Contrast readout directions compare a target token’s LM-head direction with an average of negative alternatives. The coordinate-token version uses the sample-specific numeric token, and its dimension rankings depend partly on the negative-token pool.
A.5 Contrast readout directions
- The contrast readout direction points toward the target token relative to the average competing-token direction.The scalar for each hidden dimension is its contrastive logit-facing direction.Source: §A.5 · p. 18
- Jaccard overlap is at least .95 for the main-pool replica, but .24–.61 for alternative-pool Top-k sets, showing partial negative-pool dependence.The authors treat the negative pool as part of the ranking specification and ground causal claims in held-out interventions rather than rank identity alone.Source: Table 9 · p. 19, §A.5 · p. 19
- For coordinate-token rankings, the target is the sample-specific coordinate token; bbox_2d rankings instead use stable field-token positions.Source: §A.5 · pp. 18–19
A.6 Corruption-derived ranking
This section distinguishes clean-readout leverage from object-mask and partial-occlusion recovery using intervention-based diagnostics. It treats endpoint rankings and coordinate-token pathways as useful but incomplete circuit-tracing references, with recovery effects varying by restoration budget and dimension-set identity.
A.6 Corruption-derived ranking
- Corruption-derived rankings aggregate object-mask clean-to-corrupt endpoint displacements, identifying dimensions with recurring recovery leverage rather than clean-readout output-interface leverage.Aggregation uses mean scores when raw scores are available, or average sample-level rank position when only orderings are retained.Source: §A.6 · pp. 19–20
A.7 Flooring motivation and readout-ranking score
- Top-k flooring recovers more of the full final-site gain than size-matched random sets, while complement flooring removes or reverses the gain once the withheld Top-k set is large enough.At very small k, complement sets can outperform Top-k because they retain most dimensions; the ranking is an endpoint reference, not evidence that endpoint-ranked dimensions suffice for object-mask repair.Source: §A.7 · pp. 20–21
A.8 Partial-occlusion corruption
- Partial occlusion masks an in-box patch whose area matches the specified fraction, leaving part of the target visible and reusing each sample’s corruption across conditions.Reported sensitivity analyses use θ = −∞, making floored restoration equivalent to clean endpoint replacement.Source: §A.8 · p. 21
A.9 Dimension sets and intervention operators
- At 40% occlusion, coordinate-token sets are negative at k = 1000 but nonnegative at k = 1280, while corruption-derived sets are reliably harmful at k = 1280.At 40% occlusion and k = 1280, bottom-ranked dimensions are the strongest positive condition; random restoration is positive at k = 1000 for 20%–40% occlusion but nonpositive at k = 1280.Source: Table 10 · p. 22, §A.9 · p. 22
A.10 Layer-resolved selective flooring
- Layer-resolved selective flooring moves a fixed ranking across decoder layers and the final site to locate where ranked and non-top dimensions become load-bearing.The study compares full, Top-k, complement, and random flooring conditions at k = 500, 1000, and 1280; disruptive complement flooring indicates load-bearing non-top dimensions.Source: §A.10 · pp. 22–23
A.11 k-scaling endpoint recovery
- Endpoint-recovery k-scaling compares ranked, random, complement, and full clean-value replacement to test whether recovery is ranking-enriched or capacity-driven.A positive Top-minus-random gap indicates ranking-enriched recovery, whereas Top/random/bottom parity indicates dependence mainly on the number of replaced dimensions.Source: §A.11 · p. 23
A.12 Local-set endpoint recovery
- Local-set recovery tests whether fixed, split-local, or sample-specific dimension sets improve on global rankings across k = 500, 1000, and 1280.The sample-specific corruption-derived set selects dimensions disrupted by that sample’s visual corruption; the population corruption-derived set selects dimensions most often disrupted across the population.Source: §A.12 · p. 24
A.13 Edge-attribution patching under partial occlusion
- Edge-attribution patching evaluates whether coordinate-token endpoint objectives isolate behaviorally important attention and MLP edges under partial occlusion; it is a path-level validation, not a recovery operator.The diagnostic scores candidate edges using coordinate-token Top-k, complement, and random half-width endpoint subspaces on shared occlusion forward passes.Source: §A.13 · pp. 24–25
- Under partial occlusion, coordinate-token Top-k edges cause only an additional −.063 R50 drop from 100 to 200 edges, versus −.163 for complement and −.134 for random edges.At 100 edges, coordinate-token Top-k gives the largest R50 drop; its circuit is highly specific to J∆ but has limited recall for the full detection circuit.Source: §A.13 · pp. 25–26
- Coordinate-token Top-k MLP edges peak at layer 35, while complement and random-k peak at layer 32 and spread more mass across layers 32–35.The coordinate-token attention circuit has a weaker layer-24-to-34 dominance ratio (1.80×) than complement (2.10×) or random-k (2.01×); the structural pattern supports concentration near coordinate-generation pathways versus broader late-layer detection routes.Source: §A.13 · pp. 26–27, Figure 2 · p. 26
A.14 Quasi-layer RMSNorm control
- Pre/post RMSNorm repair-ranking overlap is .942 ± .008 at k = 500 and .970 ± .003 at k = 1280, while VJP transport does not close the endpoint-repair gap.Endpoint-repair overlap changes by only +.008 at k = 500, +.001 at k = 1000, and −.004 at k = 1280, indicating the dissociation is already present pre-norm; this is a ranking-comparison control, not a pre-RMSNorm intervention.Source: §A.14 · pp. 27–28, Table 12 · p. 28
A.15 Floored-clean endpoint recovery ablation
- Thresholded clean replacement lowers every reported corruption-derived R50 cell versus primary clean endpoint replacement, with a larger fixed-population versus sample-specific gap at k = 1000.Clean endpoint replacement nearly closes that gap, so thresholding the inserted value can limit recovery independently of index-set quality.Source: §A.15 · pp. 28–29
A.16 Reporting and interpretation
- The reporting convention calls a ranking recovery-enriched only when matched-size Top-k sets outperform random and bottom-ranked controls and withholding those dimensions disproportionately blocks endpoint recovery.Endpoint rankings otherwise remain readout-informative but recovery-incomplete, preserving their value as circuit-tracing references without treating them as sufficient explanations of visual-corruption repair.Source: §A.16 · p. 29