PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning

Yingjin Song, Denis Paperno, Albert Gatt

arXiv:2609.35486v1cs.CLcs.CV

TL;DR

Relative-position reasoning can look accurate on individual queries while remaining inconsistent when object locations or query roles change, and its supporting representations are not well understood. The paper traces location and role representations across visual and textual inputs using mechanistic analyses and targeted interventions. Steering directions estimated on synthetic scenes improve accuracy and paired consistency on natural-image benchmarks in most settings without retraining.

  • Problem

    Instance-level accuracy can mask inconsistencies under object-location swaps or target/reference reversals, while the representations supporting relative-position reasoning remain poorly understood.

    Source: §1 · p. 1

  • Method

    The paper traces source- and query-side location representations and target/reference roles across visual and textual settings using activation patching and targeted interventions.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Steering along role directions estimated on synthetic scenes improves accuracy and paired consistency on natural-image benchmarks in most settings without retraining.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    Paired behavioral tests combined with mechanistic interventions clarify how models represent and use spatial information, with the aim of guiding efforts to improve relational-reasoning consistency and robustness.

    Source: §7 · p. 10

    Mechanistic analyses use controlled two- and three-object scenes with fixed grid positions and templated queries; freer layouts, open-ended phrasings, and other relations remain future extensions.

    Source: LIMITATIONS · p. 10

Abstract

from arXiv · show

High instance-level accuracy can mask inconsistencies in spatial reasoning when objects exchange positions or their roles are reversed in the query. The internal representations supporting relative-position reasoning remain poorly understood. We investigate two complementary components of this process: tracking object locations in the input and representing their query roles. Across three VLMs with visual or textual inputs and their language-model backbones, activation patching reveals a staged progression from early-layer source representations through intermediate-layer query-object representations to late-layer answer states. Targeted interventions further establish causal links along this progression: manipulating source-side representations shifts location information at query-object mentions and ultimately alters relation predictions. Beyond object-location information, we also identify a stable query-side direction associated with the roles of the two objects in the comparison. Steering along directions estimated on synthetic scenes generalizes to natural-image benchmarks, improving accuracy and both forms of paired consistency in most settings without retraining. Our findings reveal complementary components of relational reasoning across visual and textual settings and show how targeted interventions can improve the consistency of models' behavior.

1 INTRODUCTION

Relative-position reasoning requires recovering object locations and assigning target/reference roles, yet instance-level accuracy alone does not establish consistent behavior under changes to locations or roles. The paper examines these components across visual and textual inputs using paired tests and mechanistic interventions.

1 INTRODUCTION

  • Spatial reasoning is evaluated primarily through benchmark accuracy, while the representations underlying the task and their consistency and robustness also warrant examination.The paper frames spatial reasoning as relevant to visual, linguistic, and embodied environments.Source: §1 · p. 1
  • A correct answer to one query may fail under either an object-location swap or a target/reference reversal, which separately probe location recovery and role assignment.Both perturbations reverse the correct relation.Source: §1 · p. 1
  • The study traces source-to-query location information and tests whether target/reference roles shape relation predictions across VLM+image, VLM+text, and LLM+text settings.The methods include activation patching, source-to-query interventions, and destructive and positive steering of role directions.Source: §1 · p. 2
  • Steering along role directions estimated on synthetic scenes improves accuracy and both forms of paired consistency on What’sUp and COCO-Spatial in most settings without retraining.The paper also reports that high instance-level accuracy does not guarantee relational consistency.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work uses mechanistic tools to localize task-relevant representations in language models and VLMs, and studies spatial reasoning across visual and textual settings. This paper connects these lines by tracing location representations and target/reference roles across matched input conditions.

2 RELATED WORK

  • Mechanistic studies have used probing, activation patching, sparse autoencoders, and steering in language models, with related tools adapted to analyze VLMs.The paper applies this approach to how source-side spatial evidence reaches query-side location IDs and interacts with role binding.Source: §2 · p. 3
  • Prior spatial-reasoning work reports limited robustness in VLMs and identifies visual-attention, modality-balance, and fine-grained spatial-detail issues, while text-only LLM work has mainly tested spatial inference from language.Mechanistic VLM studies have identified object-word spatial IDs, visual-token spatial information, and a query-token-mediated pathway.Source: §2 · p. 3
  • The paper connects visual and textual analyses by tracing location representations in VLMs and their LLM backbones, linking source- and query-side location IDs, and identifying a separate target/reference role direction.Source: §2 · p. 3

3 TASK SETUP AND BEHAVIOURAL BENCHMARKING

The paper benchmarks relative-position judgments across matched visual and textual inputs, using location swaps and target/reference reversals to test paired consistency. VLM+image leads on all reported metrics, while paired tests reveal inconsistencies not evident from original-query accuracy alone.

3 TASK SETUP AND BEHAVIOURAL BENCHMARKING

  • The paired tests swap the queried objects’ locations or reverse their target/reference roles, keeping the queried axis and answer-option order fixed while reversing the ground-truth relation.For three-object scenes, the distractor remains unchanged during a location swap; text descriptions are updated to reflect the swapped layout.Source: §3 · p. 3
  • The benchmark uses two- and three-object Synthetic scenes and What’sUp subsets A and B across VLM+image, VLM+text, and LLM+text conditions.Each paired-consistency score requires both answers to match their respective ground truths.Source: §3 · p. 3
  • VLM+image leads on all three metrics for every model on both benchmarks, and five models achieve perfect accuracy and paired consistency on Synthetic.Figure 2 presents the benchmarking results; numerical results are provided in Table 6.Source: §3 · p. 3, Figure 2 · p. 4
  • On What’sUp, LLaVA-1.6-Mistral VLM+image has 88.85% accuracy, versus 81.03% target–reference reversal consistency and 78.31% location-swap consistency.On Synthetic, the Vicuna backbones of LLaVA-1.5 and LLaVA-1.6-Vicuna reach 50% accuracy by nearly always answering left or above, but score 0% on both consistency metrics.Source: §3 · p. 4

4 LOCALIZING SOURCE, QUERY AND ANSWER-STAGE REPRESENTATIONS

Activation patching localizes representations that influence answer preferences under location swaps and target/reference reversals. The observed pattern progresses from source-side evidence toward query-object representations and late answer states, with recovery sites varying by input condition.

4 LOCALIZING SOURCE, QUERY AND ANSWER-STAGE REPRESENTATIONS

  • The patching analysis replaces a semantic token group’s corrupted-run residual-stream states with corresponding clean-run states and measures recovery of the clean-answer preference.The analysis uses correctly answered clean–corrupt pairs to localize representations supporting correct judgments.Source: §4 · p. 4
  • Under location swaps, clean-answer recovery shifts from source tokens in early-to-middle layers to query-object tokens in middle layers and the final token in late layers; role reversals are recovered mainly by query-object and late final-token patches.Source recovery remains near zero under role reversal because source tokens precede the query and have identical states across the pair under causal attention.Source: Figure 3 · p. 5, §4 · p. 5
  • For VLM+image location swaps, patching all visual tokens nearly fully restores the clean answer, while object-strip patches outperform object-bounding-box patches.For text inputs, scene-description object mentions are the strongest source-side group, peaking at about 0.95 in VLM+text and 0.73 in LLM+text; joint target/reference query-span patching recovers answers under role reversal.Source: §4 · p. 5
  • VLM+text generally shows higher recovery from scene-description and query-object patches, whereas relation-option patches have larger effects in LLM+text.Source: §4 · p. 5

5 EXTRACTING AND INTERVENING ON OBJECT-CENTERED LOCATION IDS

The section extracts object-centered location IDs at source and query tokens, then tests their predictive and causal roles. Source-state patching shifts query-side location information and answer preferences, while query-side ID steering changes relation predictions.

Location ID extraction

  • Location IDs are location-label prototypes formed from attribute-centered object states, with source and query sites estimated independently but linked by shared labels.Source states use image bounding-box tokens or text description spans; query states use target or reference mentions.Source: §5 · p. 6
  • Location IDs support held-out location prediction and query-side relation-sign prediction, outperforming shuffled-label and orthogonal-axis controls.Figure 4 also visualizes prototype geometry along axes defined by endpoint-ID contrasts.Source: §5 · p. 6, Figure 4 · p. 6

Source-to-query transfer

  • Patching corrupted source states shifts query-side location scores and answer preferences; full visual-token patches have larger effects than localized bbox+strip patches.Description-side object patches also shift scores and induce answer flips in both text-input conditions.Source: Table 1 · p. 6, §5 · p. 7

Query-side location-ID steering

  • The query-side intervention swaps the target and reference objects’ source-scene spatial labels, then adds their ID difference across each query-object token span.The input remains unchanged, and the target/reference roles stay fixed.Source: §5 · p. 7
  • True query-side location-ID steering produces the strongest, most consistent belief flips, concentrated in middle layers and weaker under random, shuffled-ID, and orthogonal controls.The effect persists across VLM+image, VLM+text, and LLM+text settings.Source: §5 · p. 7, Figure 5 · p. 7

6 TARGET-REFERENCE ROLE DIRECTION STEERING

The section identifies a stable target/reference role direction in query-object states and tests its causal contribution to relation predictions. Steering with synthetic-scene directions improves paired consistency on natural-scene benchmarks.

Role-contrast geometry

  • The joint target/reference role contrasts form a stable hidden-state direction, with held-out alignment and norm-ratio metrics remaining high across models and conditions.The direction averages query-object states for the same object when it is target versus reference across TF and RF templates.Source: §6 · pp. 7–8, Figure 6 · p. 8

Role-direction intervention

  • At α = 1, destructive query-object-span updates disrupt answer margins more along the joint role direction than along matched orthogonal controls, especially in early-to-middle layers.Within fixed layer bands, the plotted gaps become more negative as α increases under both TF and RF templates.Source: §6 · p. 8, Figure 7 · p. 8

Role-direction steering

  • Joint role-direction steering improves reversal consistency in all nine What’sUp-A settings and all nine COCO-Spatial settings without retuning; accuracy improves in most settings.On What’sUp-A, the largest gains are in Pixtral VLM+text: 4.9% accuracy, 7.8% reversal consistency, and 9.4% location-swap consistency.Source: Table 2 · p. 9, §6 · p. 9
  • Steering applies the query-specified role signal to query-object tokens, with layers and strengths selected on disjoint synthetic validation scenes and frozen for benchmark evaluation.Model parameters remain unchanged.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents object-location information and query-role representations as complementary components of relative-position reasoning. It also reports that synthetic-scene role directions improve accuracy and paired consistency on natural-image benchmarks in most settings.

7 CONCLUSION

  • The analyses support a staged progression from early source representations through intermediate query-object representations to late answer states, with interventions linking location information to relation predictions.The conclusion characterizes object-location information and query-role representations as causally relevant, complementary components.Source: §7 · p. 9
  • The findings extend evidence for causally relevant object-location representations from image-conditioned VLMs to textual inputs in VLMs and their LLM backbones.The conclusion also reports improved accuracy and paired consistency in most natural-image benchmark settings using synthetic-estimated role directions without retraining.Source: §7 · p. 10
  • The conclusion argues that paired behavioral tests combined with mechanistic interventions can clarify how models represent and use spatial information.This is presented as guidance for future efforts to improve relational-reasoning consistency and robustness.Source: §7 · p. 10

AI USE STATEMENT

The statement describes the use and review of generative AI tools in the research process.

AI USE STATEMENT

  • Generative AI supported code development and debugging, literature identification, and manuscript writing and revision; authors reviewed the AI-assisted materials and take responsibility for the final content.The statement specifies that wording and clarity were particular targets of manuscript assistance.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies where experimental and implementation details are documented and what materials are planned for release.

REPRODUCIBILITY STATEMENT

  • Appendices document data construction, datasets, model checkpoints, benchmarks, experimental procedures, metrics, software settings, and compute resources; code, configurations, and synthetic datasets will be released upon publication.The corresponding experimental sections specify activation-patching and steering procedures and evaluation metrics.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The study uses controlled synthetic data and existing benchmarks to investigate spatial reasoning mechanisms in VLMs and LLMs, without collecting human-subject data or personal information. It treats activation steering as an analysis tool while acknowledging its potential misuse.

ETHICS STATEMENT

  • The study uses controlled synthetic data and existing benchmarks to investigate mechanisms underlying spatially consistent predictions.The stated downstream relevance includes robotics and assistive technologies.Source: ETHICS STATEMENT · p. 10
  • The work collects no human-subject data, personal information, or sensitive attributes.Source: ETHICS STATEMENT · p. 10
  • Activation steering could be misused to manipulate model behavior, but the authors use it only to analyze representations and robustness.Source: ETHICS STATEMENT · p. 10

LIMITATIONS

The supplementary material bounds the study to controlled spatial tasks and details its scene construction, evaluation setup, and additional activation-patching results. Those results show condition-dependent recovery patterns across source, query, and late answer representations.

LIMITATIONS

  • The mechanistic scope is fixed-grid two- and three-object scenes with templated queries; freer layouts, open-ended phrasings, and broader spatial relations remain future work.The analyzed models are three open 7B–12B models; larger, MoE, and proprietary models remain to be examined.Source: LIMITATIONS · p. 10

A DETAILS OF DATA CONSTRUCTION

  • Synthetic scenes use fixed grid positions and color-shape object identities; the primary three-object setting uses collinear layouts and six directed pairwise questions.Text descriptions preserve object identity and spatial structure through deterministic rules and ordered variants along the queried axis.Source: A DETAILS OF DATA CONSTRUCTION · p. 15

B BENCHMARKING MODELS, DATA AND NUMERICAL RESULTS

  • The evaluation distinguishes original-query accuracy from paired reversal and location-swap consistency, with each consistency score requiring both paired answers to be correct.Synthetic text conditions pool two description variants per scene; the supplementary material points to numerical benchmarking results but provides no cell values in these passages.Source: B BENCHMARKING MODELS, DATA AND… · pp. 15–16, Table 6 · p. 17

C ADDITIONAL ACTIVATION PATCHING RESULTS

  • Source-side visual tokens dominate VLM+image recovery for location swaps, while query-side target-reference representations drive reversal recovery and late final-token states recover answers.The same pattern is reported with distractors in three-object scenes; the recovery sequence under location swaps moves from early-to-middle source tokens through middle-layer query-object tokens to the late final token.Source: C ADDITIONAL ACTIVATION PATCHING RESULTS · p. 18, Figure 9 · p. 18

D LOCATION ID VALIDATION RESULTS

Held-out tests support the validity of extracted location IDs across models and input settings, and their prototypes form spatially ordered geometries. Query-side axes also predict relative direction, while role-direction interventions show strength-dependent effects and are evaluated on held-out and natural-image settings.

D LOCATION ID VALIDATION RESULTS

  • Held-out source- and query-side location IDs yield high best-layer recovery across three models and input conditions, while label-shuffled controls remain substantially lower.This supports genuine object-position binding rather than artifacts of object identity or token statistics.Source: D LOCATION ID VALIDATION RESULTS · p. 18, Figure 12 · p. 20
  • Extracted prototypes separate along horizontal and vertical axes in two-object scenes and form ordered left/middle/right and above/center/below geometries in three-object scenes.Source: D LOCATION ID VALIDATION RESULTS · p. 18, Figure 13 · p. 21
  • Projecting the difference between query-object states onto the extracted query-side spatial axis predicts relative direction with high accuracy across input settings, unlike near-chance orthogonal controls.The result indicates that the axis supports direct comparison of object-level location information.Source: D LOCATION ID VALIDATION RESULTS · p. 19, Figure 14 · p. 22

E ROLE DIRECTION INTERVENTION RESULTS

  • Increasing role-direction steering strength generally makes the role-specific margin gap more negative, and held-out synthetic steering often improves paired consistency, especially in text-based conditions.VLM+image results are frequently saturated; synthetic-selected configurations are reused without retuning for the described COCO-Spatial and GQA-Spatial evaluations.Source: E ROLE DIRECTION INTERVENTION RESULTS · pp. 19–20

F IMPLEMENTATION DETAILS AND COMPUTE RESOURCES

  • Interventions use custom PyTorch hooks with pretrained parameters frozen; inference and GPU interventions ran on one NVIDIA A100-SXM4 GPU with 40 GB memory.The reported uncertainty estimates use paired cluster bootstrapping with 2,000 replicates.Source: F IMPLEMENTATION DETAILS AND COMPUTE… · p. 25
Loading 2609.35486v1…