PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models

Raphi Kang, Hongqiao Chen, Georgia Gkioxari, Pietro Perona

arXiv:2601.12626v1cs.CV

TL;DR

VLMs’ spatiotemporal reasoning mechanisms remain opaque, motivating a search for representations that connect visual structure with textual computation. The paper constructs and causally tests a linear spatial-ID model, extends it to video temporal IDs, and reports diagnostic and learning uses alongside scope limitations.

  • Problem

    The paper investigates how VLMs combine visual or geometrical structure with textual representations to perform spatial and temporal reasoning.

  • Method

    The paper conducts mechanistic analyses of autoregressive VLMs, extracting text-anchored spatial IDs and testing them through targeted belief steering and related interventions.

  • Results

    The analyses show that spatial IDs mediate models’ beliefs about object locations, while analogous temporal IDs mediate temporal reasoning in video models.

  • Takeaways & Limitations

    Spatial and temporal IDs provide mechanisms for interpreting VLM reasoning and tools for diagnosing limitations and improving existing models.

  • Takeaways & Limitations

    The work is limited to simple spatial or appearance-based temporal queries and models up to 14B parameters.

Abstract

from arXiv · show

Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial structure must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally explain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding \textit{spatial IDs} to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions we demonstrate that these IDs, which are ubiquitous across the model, can systematically mediate model beliefs at intermediate VLM layers. Additionally, we find that spatial IDs serve as a diagnostic tool for identifying limitations in existing VLMs, and as a valuable learning signal. We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotemporal ID mechanism, we elucidate a previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models. We release our code for reproducibility: https://github.com/Raphoo/linear-mech-vlms.

1 INTRODUCTION

The paper asks how VLMs perform spatial and temporal reasoning despite opaque internal mechanisms, proposing that linear spatial IDs bind visual structure to textual representations. It tests this account through interventions and extends it to temporal IDs in video models.

  • VLMs must resolve references, locate entities, assess spatial relations, and make categorical decisions for visual question answering.
  • The paper asks whether emergent structured spatial reasoning in VLM internals can be modeled linearly.
  • Spatial IDs are text-anchored latent structures that bind visual elements to object tokens for linguistic reasoning about space.
  • Targeted interventions show that perturbing only spatial IDs can manipulate model beliefs, supporting their causal role in reasoning.
  • The analysis uses spatial IDs to study depth-expression limitations and systematic LLaMA/LLaVA failures, while testing spatial-ID guidance for finetuning.
  • Video models use analogous linear temporal IDs that can drive temporal reasoning.

2 EMERGENT STRUCTURE IN SPATIAL VISUAL REASONING

The analysis identifies a layer-specific transfer of spatial information from visual patches to object-word tokens, then models that information as approximately linear spatial IDs. Causal and theoretical analyses support spatial IDs as a simple component of VLM spatial reasoning while acknowledging broader circuit complexity.

  • 2.1 TRACKING INFORMATION FLOW DURING REASONING: The experiments test whether localized intermediate representations carry spatial information by swapping selected activations between original and mirrored image-text inputs.The intervention compares all text tokens, all image patches, and object-word tokens only, and measures the resulting belief shift.
  • 2.1 TRACKING INFORMATION FLOW DURING REASONING: Interventions reveal modality-specific information flow: visual patch effects fade with depth, text-token effects grow later, and object-word swaps matter in intermediate layers.Mirror swapping produces distinct belief swaps, whereas attribute swapping is noisy.
  • 2.2 EMPIRICAL DERIVATION OF SPATIAL IDS: Spatial IDs are text-anchored latent structures that bind visual spatial information to object-word activations for subsequent linguistic reasoning.The paper extracts IDs by averaging object-conditioned activations across images and objects, then derives horizontal and vertical directions from grid-aligned differences.
  • 2.2 EMPIRICAL DERIVATION OF SPATIAL IDS: Across 11 VLMs, spatial IDs are extracted from 4 × 4 object grids, with projected ID coordinates tracking object locations and spatial axes analyzed for separability.The study reports image invariance and orthogonal spatial-axis analyses alongside the grid extraction procedure.
  • 2.3 THEORETICAL SKETCH OF SPATIAL IDS: A simplified theory describes spatial IDs as approximately linear transformations of shared positional features written into object tokens by attention.A low-rank fit from positional encodings to spatial IDs explains most variance, with rank-3 achieving R2 ≳0.85 in the cited analysis.
  • 2.3 THEORETICAL SKETCH OF SPATIAL IDS: The authors caution that real VLM reasoning circuits may be more varied, distributed, and nonlinear, so spatial IDs may capture only one component.The framework is presented as a simplified but useful account of information transfer into text tokens.

3 SPATIAL IDS MEDIATE MODEL BELIEFS

The paper tests whether spatial IDs causally mediate VLM beliefs by adding or subtracting them from internal activations. Interventions change spatial, distance, and relational answers, with spatial-ID steering outperforming noise and having strongest effects at modality-alignment layers.

  • 3 SPATIAL IDS MEDIATE MODEL BELIEFS: The causal test perturbs object-token activations by adding one spatial ID and subtracting its mirrored counterpart, then measures changes in output beliefs.The intervention is applied at selected layers and preserves the approximate activation norm through scaling.
  • 3 SPATIAL IDS MEDIATE MODEL BELIEFS: Spatial steering is evaluated on real COCO images using log probabilities of directional answer tokens, with extensions to distance and three-way spatial relations.The experiments use queries such as whether one object is left or right of another and test near/far and sandwiching judgments.
  • 3 SPATIAL IDS MEDIATE MODEL BELIEFS: Steering effects peak at modality-alignment layers, and rightmost or leftmost IDs increase beliefs in the corresponding rightward or leftward answers.The dataset-level plots report changes in log probability across ID locations.
  • 3 SPATIAL IDS MEDIATE MODEL BELIEFS: 64.6% median binary belief swaps occur with spatial-ID steering versus 29.5% with noise, a 43.6% increase in the reported swap rate.A belief is counted as swapped when the relative likelihood of the ground-truth answer and its opposite reverses after intervention.

4 SPATIAL IDS FOR UNDERSTANDING AND IMPROVING IMAGE VLMS

Spatial IDs expose how image VLMs represent and use spatial information, revealing depth correlations, architectural failure points, and a learning signal for faster generalization.

  • Overview: Spatial IDs connect visual objects to linguistic reasoning, enabling analysis of how image VLMs represent spatial relationships.The section uses these representations to study both failure mechanisms and model improvement.
  • 4.1 Depth Representation: Height-steered spatial IDs also shift beliefs about depth, with “above” aligned toward “front” and “below” toward “behind” in LLaVA.Overlapping spatial-ID locations for these word pairs indicate a strong height-depth correlation.
  • 4.2 Diagnosing VLMs: Incorrect spatial-ID deviations correlate with wrong answers in both LLaVA and LLaMA, indicating that linguistic reasoning was not the primary failure source.Negative ID margins indicate extracted spatial IDs opposing ground truth.
  • 4.2 Diagnosing VLMs: Image masking distinguishes likely bottlenecks: LLaVA shows faulty object detection, whereas LLaMA’s failures appear after the vision encoder during information integration.The authors describe these diagnoses as preliminary rather than universal explanations of model failures.
  • 4.3 Improving VLMs: Stronger spatial-ID steerability tracks higher COCO-spatial accuracy, and adding spatial-ID loss lets Qwen2-2B reach 91% accuracy at 3.2k steps, a net 6% gain over control.Both models later overfit after peaking near 90% validation accuracy on the simplistic synthetic training data.

5 TEMPORAL IDS IN VIDEO MODELS

Video VLMs exhibit temporal analogues of spatial IDs: frame-order information is encoded in object-word activations and can be extracted and causally steered.

  • Temporal ID Framework: The temporal analysis tests whether video models use linearly separable temporal markers on object-word activations.The experiments repeat the spatial-ID framework for the temporal dimension.
  • Temporal Mirror Swapping: Mirroring videos by reversing frame order produces temporal effects when text, image-patch, or object-word activations are swapped.The observed error bound is noisier than in spatial LLaVA because LLaVA-Video follows response formats less consistently.
  • Temporal ID Extraction: In synthetic eight-frame videos, “before” projects closer to earlier frames than “after,” supporting temporal ID extraction.The temporal vector is derived using the same formal procedure as the spatial analysis.
  • Causality of Temporal IDs: On naturalistic MVBENCH videos, later temporal IDs steer beliefs toward “after,” while earlier IDs steer them toward “before.”This provides causal evidence that temporal IDs are controllable in real video inputs.
  • Cross-Model Evidence: Temporal IDs also appear in VideoLLaMA3 and Qwen2.5, including settings with textual timestamps or explicit MRoPE time IDs.The authors report that spatiotemporal IDs can emerge without explicit positional encoding.

6 RELATED WORK

The work builds on mechanistic interpretability methods showing linear structure and internal reasoning in language and vision-language models.

  • Mechanistic Interpretability: Mechanistic interpretability studies use circuit tracing, sparse autoencoders, linear probing, and activation patching to investigate model internals.The Linear Representation Hypothesis motivates viewing concepts as linearly encoded in latent activations.
  • Vision-Language Models: Prior VLM research identifies linearity in latent spaces and separates visual question answering into image-focused and text-focused stages.Other studies adapt logit-lens and attention-tracking methods to uncover VLM circuits.

7 CONCLUSION, LIMITATIONS, & FUTURE WORK

The paper presents spatiotemporal IDs as a linear account of visual reasoning and extends the analysis toward diagnosis and improvement, while limiting conclusions to relatively simple queries and models up to 14B parameters.

  • Conclusion: Spatiotemporal IDs provide a linear model for visual reasoning, and causal analyses show they mediate beliefs about object locations in space and time.The authors position this mechanism as a basis for improving existing VLMs.
  • Limitations and Future Work: The analysis is limited to simple spatial queries and appearance-based temporal queries, leaving more complex open-ended reasoning for future work.The authors also note that explicit temporal guidance at larger scale remains an open use case.
  • Limitations and Future Work: Extraction and steering are limited to models of at most 14B parameters because of compute constraints.Larger models may use analogous or distinct visual-reasoning mechanisms, which the paper does not establish.

REPRODUCIBILITY

The paper provides detailed experimental information and releases code to support reproducibility.

  • REPRODUCIBILITY: Experimental details, ablations, counterfactual trials, and results for all considered models are provided in the appendices.The authors direct readers to Sections A–D for these materials.
  • REPRODUCIBILITY: Code for all experiments is publicly available.The repository is hosted at https://github.com/Raphoo/linear-mech-vlms.

LINEAR MECHANISMS FOR SPATIOTEMPORAL REASONING IN VISION LANGUAGE MODELS –SUPPLEMENTARY MATERIAL–

The supplementary material documents token handling, intervention procedures, and synthetic data used to extract spatial and temporal IDs. It also illustrates the extraction pipeline for spatial IDs and the examples used to test model beliefs.

  • Token handling: Tokenization differs across model families, including single-token and multi-token representations of object words.For multi-token words, the final token is used as the representative object token.
  • Logit probabilities: Model likelihoods for competing answers are evaluated using the log probability of the corresponding output token.The procedure follows each model family’s tokenization scheme.
  • Activation patching: Activation patching replaces selected original intermediate activations with those from mirror-swapped inputs.Hooks collect activations for both cases and perform replacements at selected indices.
  • Synthetic data generation: Spatial-ID extraction uses 90 object pairs, four object sizes, and 86,400 synthetic images generated from OBJAVERSE renders.Objects are projected in pairs onto random backgrounds across grid locations.
  • Synthetic data generation: Temporal-ID extraction uses five unique object pairs in 61 temporal arrangements, with centered objects for baseline extraction.Three horizontal positions are added to verify spatial-temporal disentanglement.
  • Spatial-ID extraction: Spatial-ID extraction isolates an object-word activation’s shared lexical component and separates it from spatial localization.The resulting linearly bound spatial ID is obtained from the residual spatial component.

A.4 COLOR-BINDING REASONING EXPERIMENTS

The supplementary experiments test whether spatial IDs mediate color-based reasoning, how interventions affect model beliefs, and where spatial information is stored. They also examine oracle injections, spatial-axis structure, and spatial-temporal separation.

  • Color-binding reasoning: Swapping color-word activations changes model beliefs more than swapping non-color word activations across 100 images.The result supports spatial-ID mediation beyond direct spatial queries.
  • Color-binding reasoning: Spatial-ID interventions alter model beliefs when applied to color-related words, while swapping an object token with unchanged location has no effect.The contrast indicates that color words can store location-specific spatial information.
  • Oracle injection: 13.4% above baseline accuracy is achieved by LLaVA when oracle spatial IDs are injected at layer 1.Intermediate-to-later-layer intervention has little effect in LLaVA.
  • Learning signal: Around 6% absolute validation improvement beyond baseline reaches 90% accuracy in under 2.8k steps with spatial-ID loss.This result is reported on the COCO-spatial naturalistic validation set.
  • Information storage: Spatial IDs from non-object words partially steer model beliefs, but most spatial-ID information remains concentrated in object-word tokens.The non-object-word effect is attributed to some semantic information being distributed across token activations.
  • Axis structure: Spatial and temporal axes are well separated in the tested LLaVA-Video setting, while some models show correlated horizontal and vertical axes.The spatial-temporal result uses objects placed in eight frames and three locations.

D.1 SCALING ANALYSIS FOR SPATIOTEMPORAL ID EXTRACTION

Scaling analyses show that spatial IDs remain recoverable across extraction-set sizes, prompt formulations, and object sizes. Spatial axes explain at least roughly half of the variance in the tested settings, although object-size changes reduce explained variance.

  • Extraction-set scaling: At least 50% of variance is explained by spatial axes even with as few as nine object pairs.IDs extracted using 6, 20, and 56 object-pair cases still converge into tight color-wise clusters at intermediate layers.
  • Prompt wording: Spatial information is extractable from object words under both spatial and plain prompt formulations.The prompt wording changes from a left-or-right query to an object-presence query.
  • Object-size scaling: At least 50% of variance remains explained across object sizes, despite a 10–20% drop when projecting onto axes learned from medium-sized objects.The tested diameters are 80px, 128px, and 176px.

E.1 INFORMAL PROOF FOR SPATIAL ID EMERGENCE

The analysis argues that positional encodings generate universal spatial IDs through attention, which can be linearly aligned with spatial coordinates and language readouts.

  • A patch vector is modeled as semantic content plus a shared positional basis mapped into model space and a small deviation.The analysis assumes objectness makes attention concentrate on the unique patch containing the queried object.
  • Attention updates bind the object token to a location-dependent component that is independent of object identity after averaging over objects.The fixed matrix WoutWVP maps the shared positional basis into a universal spatial ID.
  • If the language readout difference between “left” and “right” aligns with the positional x-coordinate, the model predicts the corresponding spatial word.
  • Multi-head and multi-layer contributions accumulate linearly in the residual stream, with alignment bands marking layers where the spatial signal is strongest.
  • Rank-3 regression relates positional encodings to spatial IDs with R2 ≥0.85, suggesting horizontal, vertical, and radial components are preserved.The relationship is estimated by least-squares regression, optionally constrained through truncated singular value decomposition.
Loading 2601.12626v1…