PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Depth but Choosing Cues in Vision-Language Models

Zhengtao Zou, Pekka Marttinen

arXiv:2610.17976v1cs.AI

TL;DR

This paper asks whether VLMs use depth information or image-plane cues when judging which object is closer. It tests cue conflicts, reads depth from object representations, and intervenes on those representations; judgments follow misleading cues even though depth remains decodable and can support correction.

  • Problem

    It is unclear whether VLMs judge which object is closer using depth information or simpler image-plane cues such as apparent size and vertical position.

    Source: §1 · p. 1

  • Method

    The paper tests controlled depth–cue conflicts, probes queried-object and answer states for depth information, and uses object-level write-back to test its influence on decisions.

    Source: §1 · pp. 1–2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    Across model families, judgments follow misleading image-plane cues, while correct depth order remains decodable and object-level write-back can improve conflict decisions without parameter updates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VLMs can retain useful depth information while competing image-plane cues exert greater influence on their answers.

    Source: §1 · p. 2

    The analysis is limited to pairwise depth judgments and apparent size and vertical position; real-world images contain additional depth cues, and intervention effects vary across model families and domains.

    Source: §6 · p. 9

Abstract

from arXiv · show

Do vision-language models use depth information to judge which object is closer, or rely on image-plane cues? We investigate this question through controlled depth--cue conflicts, representation analysis, and interventions. Among the many image-plane cues relevant to depth perception, we focus on apparent size and vertical position. We compare depth judgments under cue-consistent and cue-conflict conditions in both rendered scenes and real-world photographs. Across model families and scales, depth judgments systematically shift toward misleading cues. Yet the correct depth order remains decodable from the queried objects' visual representations, including when the model answers incorrectly. Image and category controls support interpreting this signal as image-dependent depth information beyond the two target cues. Strengthening the model's own depth readout through object-level write-back can correct conflict decisions without updating model parameters or using ground-truth depth annotations. Interventions show that the effect of this correction depends on the direction and target tokens of the written signal, while its strength varies across architectures and domains. Together, these findings reveal a mismatch between a model's ability to represent depth and use this information when deciding the answer. VLMs can retain useful depth information while allowing competing image-plane cues to exert greater influence on their answers.

1 INTRODUCTION

This paper asks whether VLMs judge object depth from depth information or from image-plane cues such as apparent size and vertical position. Across models, judgments follow misleading cues even though the correct depth order remains represented and can guide object-level write-back.

1 INTRODUCTION

  • The study focuses on why a closer-object judgment can be unreliable despite VLMs’ strengths elsewhere, testing whether models use depth or simpler predictive image-plane cues.The paper highlights apparent size and vertical position as cues that often predict front–behind relationships in natural images.Source: §1 · p. 1
  • Across eleven VLMs from five families, depth judgments systematically follow misleading size and vertical-position cues when they conflict with true depth.Neither model scaling nor spatial fine-tuning eliminates this pattern.Source: §1 · pp. 1–2
  • The correct depth order remains linearly decodable from queried-object visual contrasts, including when the model answers incorrectly.Image and category controls support interpreting this signal as dependent on the current image.Source: §1 · p. 2
  • Object-level write-back, oriented by the model’s own depth readout, can correct conflict decisions without updating VLM parameters.Matched controls test whether the effect depends on the direction, target tokens, and write-back configuration.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses spatial shortcuts, internal spatial representations, and methods for improving spatial reasoning. This paper distinguishes itself by testing queried-object depth representations and their influence under controlled cue conflicts.

2 RELATED WORK

  • Min et al. (2026) report sensitivity to perspective cues, particularly vertical position; this paper controls apparent size and vertical position to test whether depth remains represented during cue-supported errors.The analysis examines the internal basis of the behavior.Source: §2 · p. 2
  • Liao et al. (2026) distinguish grounded, prior-driven, and inverted depth representations, while Kang et al. (2026) identify steerable linear spatial directions.This paper instead reads depth from queried-object contrasts and tests their influence under controlled cue conflicts.Source: §2 · p. 2
  • SpatialVLM (Chen et al., 2024) and SpatialRGPT use additional supervision, while contrastive decoding methods modify generation without updating model weights.This paper uses a frozen model to test available depth information when it conflicts with apparent size or vertical position.Source: §2 · p. 2

3 CONTROLLED DIAGNOSIS OF DEPTH–CUE COMPETITION

The study tests pairwise depth judgments when apparent size and vertical position agree or conflict with true depth. Across model families, conflict reduces accuracy, and stronger misleading cues increasingly pull choices away from the correct depth order.

3.1 EXPERIMENTAL SETUP

  • The setup compares consistent, conflicting, and decorrelated cue–depth relations in rendered scenes and photographs paired with sensor depth.Rendered scenes independently vary physical size and image-plane position while preserving depth order; natural-image scene properties are not independently controlled.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 VLMS FOLLOW 2D CUES WHEN THEY CONFLICT WITH DEPTH

  • Vertical-position conflict causes a larger accuracy drop than size conflict in eight of eleven models, and the same pattern appears in natural photographs with smaller gaps.In the real-world image pool, lower objects are closer in 76.3% of pairs exceeding the vertical-separation threshold, versus 58.1% for larger objects exceeding the size-difference threshold.Source: §3.2 · pp. 3–4
  • Depth accuracy drops sharply across model families when either image-plane cue conflicts with true depth.On rendered scenes, Qwen3-VL-8B reaches 42.8% under vertical conflict and LLaVA-OneVision-1.5-8B reaches 7.4%; spatial fine-tuning does not remove the effect.Source: §3.2 · p. 3, Figure 3 · p. 4
  • With true depths fixed, stronger misleading size or vertical-position cues progressively pull choices away from the correct depth order.The curves establish graded cue dependence, not a comparison of raw slopes across cues measured in different units.Source: Figure 4 · p. 4, §3.2 · p. 4

4 REPRESENTATION OF DEPTH ORDER

Queried-object contrasts expose a depth-order signal across layers, while answer-state probes show that its accessibility differs from object representations and the model’s final decision.

4.1 LINEAR READOUT OF THE DEPTH TRACE

  • Mean-pooled queried-object contrasts train a linear depth-order probe; the intervention uses an offset-free projection onto its depth direction as the depth trace.A positive trace favors object A as closer, and a negative trace favors object B.Source: §4.1 · p. 5
  • On Qwen3-VL-8B, object-level readout probe accuracy stays above 97.3% at every language layer above embeddings on rendered scenes and ranges from 82.9% to 91.7% on real-world photographs.Source: §4.1 · p. 5
  • Qwen3-VL-8B readout accuracy falls from 82.6% to 40.3% with a grey image, while unseen category combinations reach 84.4% versus 84.5% overall.Mismatched-image controls also sharply reduce accuracy, supporting dependence on the current image beyond apparent size and vertical position.Source: §4.1 · p. 6
  • Depth becomes linearly accessible at different points across architectures: Qwen3-VL-8B reaches 99.7% at language-model input embeddings, while Gemma-4-12B nears ceiling after its first decoder block.The probes locate linear accessibility, not the first existence of depth information.Source: §4.1 · p. 6

4.2 DEPTH INFORMATION AT DECISION STATES

  • Answer-state probes predict the correct answer letter more accurately than closer-object identity, peaking at 91.3% for Qwen and 85.3% for OneVision on Render.On Real, answer-letter accuracy peaks at 75.3% and 67.2%, respectively.Source: §4.2 · p. 6
  • On rendered vertical conflicts, Qwen’s object-level readout is correct on 99.0% of examples while the model answers correctly on only 42.8%.Object-representation depth decoding remains high throughout the stack on Render; on Real it peaks mid-stack and declines toward final layers.Source: §4.2 · p. 6

5 TESTING THE INFLUENCE OF DEPTH INFORMATION

The paper intervenes on the model’s own depth direction while keeping its input and weights fixed. Write-back improves conflict decisions in some settings, but its effects vary across models and domains and depend on the intervention configuration.

5.1 WRITE-BACK INTERVENTION TO STRENGTHEN THE DEPTH TRACE

  • The depth direction is fitted on decorrelated calibration data, and layer and strength are selected on development data by maximizing the weaker conflict-gap recovery subject to a cue-consistent-accuracy constraint.The evaluation example’s depth label is not used to orient the intervention; the selected settings are frozen before evaluation.Source: §5.1 · p. 7
  • At the selected layer, opposite updates along the depth direction are applied to the two queried objects according to the sign of their pre-intervention trace.Only queried-object tokens are directly modified; the input and model parameters remain fixed.Source: §5.1 · p. 7

5.2 STRENGTHENING THE DEPTH TRACE CHANGES THE DECISION

  • On rendered scenes, Qwen3-VL-8B accuracy rises from 50.2% to 90.8% under size conflict and from 42.8% to 71.4% under vertical conflict, while cue-consistent accuracy changes from 94.4% to 99.6%.The table reports accuracy before and after intervention across rendered and natural images.Source: §5.2 · p. 7, Table 1 · p. 8
  • Intervention effects vary across architectures and domains: gains are large for some rendered and real-image settings, but mixed or negative for InternVL3 and Gemma-4 on real photographs.Cue-consistent accuracy often decreases on real photographs.Source: §5.2 · p. 8

5.3 ABLATIONS

  • Cue-avoidant prompts and predicted depth maps do not reproduce the gains; random directions and answer-token writes yield little selective improvement, while reversing orientation reverses the behavioral shift.The ablation table identifies object-level write-back as giving the largest improvement for size and vertical conflicts.Source: §5.3 · p. 8, Table 2 · p. 9
  • Write-back strengthens the relationship between the pre-intervention depth trace and answer margin, with a clearer separation on Render and a sharper transition around z = 0 on Real.The answer margin is the logit for the option naming object 0 minus that for object 1; positive values favor object 0.Source: §5.3 · p. 9, Figure 10 · p. 9

6 LIMITATIONS

The analysis is limited to pairwise depth judgments and two image-plane cues; the authors also note that effects vary across model families and domains. The paper reports that a coding agent assisted with scripts and result-artifact organization under author direction.

6 LIMITATIONS

  • The study covers pairwise depth judgments and apparent size and vertical position, leaving additional depth cues and broader spatial reasoning for future work.Real-world images contain other depth-relevant signals that are harder to isolate and may contribute to smaller observed effects.Source: §6 · p. 9
  • The causal effect varies across model families and domains, constraining how broadly the intervention findings extend.The authors identify additional depth cues and more general spatial reasoning as directions for future work.Source: §6 · p. 9

A STATEMENT ON THE USE OF AI ASSISTANCE

  • A coding agent wrote and ran scripts and organized result artifacts under the authors’ direction; experimental design and decisions remained the authors’.The authors take responsibility for the manuscript and results.Source: A STATEMENT ON THE USE OF AI ASSISTANCE · p. 11

B REPRODUCIBILITY STATEMENT

The reproducibility statement identifies code, result artifacts, checkpoint revisions, and image-level manifests in the working repository. External benchmarks reproduce the cue-conflict pattern, but the small, mixed-source conflict subset is supporting evidence rather than a standalone estimate.

B REPRODUCIBILITY STATEMENT

  • The working repository contains experimental code, result artifacts, checkpoint revisions, and manifests recording image-level pairs, labels, assignments, and cue values.Data construction and scoring, probe fitting, write-back, and controls are described in appendices D and E.Source: B REPRODUCIBILITY STATEMENT · p. 11

C.1 EXTERNAL DIAGNOSIS

  • On SpatialRGPT depth items, Qwen3-VL-8B scores 88.9% on cue-consistent items and 69.2% on conflicts, reproducing the pattern on data not constructed by the authors.Five items are near-duplicates of evaluation images, including one calibration image.Source: §C.1 · p. 11
  • The SpatialRGPT conflict subset contains only 13 mixed-source items, so it is supporting evidence rather than a standalone estimate.The benchmark also includes five near-duplicates of evaluation images.Source: §C.1 · p. 11

C.2 INTERVENTION RESULTS

The intervention helps on a small external conflict subset, is nearly neutral on pooled SpatialTunnel for Qwen3-VL-8B, and does not improve other external settings. The diagnostic datasets compare depth with size and vertical-position cues in rendered scenes and real photographs.

C.2 INTERVENTION RESULTS

  • Intervention helps the small SpatialRGPT conflict subset, is nearly neutral on pooled SpatialTunnel for Qwen3-VL-8B, and lowers accuracy on CV-Bench and Qwen3-VL-32B SpatialTunnel.Table 3 applies the intervention after fitting its direction on calibration images from each target benchmark; these negative results limit transfer beyond the diagnostic distributions.Source: Table 3 · p. 11, §C.2 · p. 11
  • Vanilla accuracy is shown across model scales, with cue-consistent conditions on the upper spokes and cue-conflict conditions on the lower spokes.The SpatialRGPT conflict spoke has 13 items; the CV-Bench cue split uses measured image cues and benchmark depth ordering.Source: Figure 11 · p. 12, §C.2 · p. 12

D.1 RENDERED SCENES

  • Rendered scenes independently sample depth, angular size, and image position, retaining other depth cues such as fog, shading, floor, and walls.The four 500-scene splits include consistent, size-conflict, vertical-conflict, and decorrelated cue conditions.Source: §D.1 · p. 12

D.2 REAL PHOTOGRAPHS

  • SUN RGB-D pairs use sensor depth within object masks and measured mask area and centroid row to define depth and image-plane cues.Pairs are selected with separation, size, position, and mask constraints; neutralized cues are sign-balanced within each conflict split.Source: §D.2 · p. 12

D.3 EVALUATION

Evaluation uses randomized answer options and measured cue conditions, with controls and probes designed to assess image-dependent depth representations. The appendices also specify dataset examples and the object-level write-back procedure and settings.

D.3 EVALUATION

  • Models are scored by comparing next-token logits for randomized single-token answers A and B; controls include patch shuffling and grey images with original masks retained.Neither control removes every possible source of spatial information.Source: §D.3 · pp. 12–13
  • SUN RGB-D cue conditions use sensor-depth order, mask area, and centroid row, with explicit thresholds for consistent, size-conflict, vertical-conflict, and decorrelated pairs.The final splits contain 305 consistent, 209 size-conflict, 144 vertical-conflict, and 289 decorrelated pairs.Source: §D.3 · p. 13

D.4 DATASET EXAMPLES

  • Figure 12 reports cue–depth agreement and variation with depth ratio, while dataset-example panels show recorded items and their depth and cue measurements.Rendered labels use scene depth; real-image labels use median sensor depth inside each object mask.Source: Figure 12 · p. 13, §D.3 · p. 13, §D.4 · p. 13

E.1 PROBE FITTING

  • Object representations average visual tokens selected by mask overlap, and probes predict depth order from the contrast between the queried objects’ representations.For real-image interventions, the direction is refitted on separate development images using pseudo-depth; evaluation uses sensor depth.Source: §E.1 · p. 16
  • On unseen category combinations, Qwen3-VL-8B achieves 84.4% sign accuracy versus 84.5% overall, while image-replacement controls test image dependence.Table 4 reports depth sign accuracy under image replacement; the grey-image and patch-shuffling controls retain some spatial information.Source: Table 4 · p. 16, §E.1 · p. 16

E.2 INTERVENTION AND SETTING SELECTION

  • Write-back edits only queried-object tokens once during the answer forward pass, strengthening the model’s existing depth ordering rather than supplying an external ordering.The update uses pre-intervention states, and subsequent decoder blocks process the edited states normally.Source: §E.2 · pp. 16–17
  • Layer and strength are selected on development data to recover both conflict gaps while limiting cue-consistent loss, then frozen before evaluation.Tables 5 and 6 report accuracy changes after write-back on rendered scenes and SUN RGB-D, respectively.Source: §E.2 · p. 17, Table 5 · p. 17, Table 6 · p. 17

E.3 COMPARISON WITH OTHER METHODS

On rendered scenes, the intervention is reported to outperform the comparison methods on both conflict splits while preserving the consistent split. Table 7 reports Qwen3-VL-8B answer accuracy.

E.3 COMPARISON WITH OTHER METHODS

  • The comparison includes prompt changes, contrastive decoders, predicted depth input, and a class-mean direction instead of the fitted probe direction.CoT changes the prompt, SpatioRoute changes prompt routing, and VCD and DoLa are contrastive decoders.Source: §E.3 · p. 17
  • The intervention is much stronger on both rendered conflict splits and preserves the consistent split.Table 7 compares Qwen3-VL-8B answer accuracy across the registered evaluation settings; the supplied passages do not give cell values.Source: §E.3 · p. 17, Table 7 · p. 18
  • Table 7 reports Qwen3-VL-8B answer accuracy (%) on rendered scenes.Each row uses its registered evaluation setting.Source: §E.3 · p. 17, Table 7 · p. 18

E.4 CONTROLS AND LIMITATIONS

Controls indicate that write-back effects depend on intervention direction and target tokens, and the effect varies across models and domains. The real-image figure reports size and vertical subsets of 157 and 111 pairs.

E.4 CONTROLS AND LIMITATIONS

  • With identical signs and magnitudes, the probe direction corrects more initially wrong answers and harms fewer initially correct answers than the mean-difference direction.The comparison uses SUN RGB-D pairs where both readouts identify the correct depth order; 95% intervals use an image-cluster bootstrap.Source: §E.4 · p. 18
  • Random directions, random signs, and writes to all image tokens or all tokens do not reproduce the object-localized result.Reversing the write-back direction reverses the behavioral effect.Source: §E.4 · p. 18
  • Figure 15’s real-image size and vertical subsets contain 157 and 111 pairs, respectively; its labels give event counts.Source: Figure 15 · p. 18
  • The intervention effect varies by model and domain: it is weak for InternVL3 on real images and harmful for Gemma-4.InternVL3-38B had nonzero readouts but no changed size-conflict predictions; changes on other real splits canceled in aggregate.Source: §E.4 · p. 18

Related papers

Loading 2610.17976v1…