S-Space: Exploring Spatial Workspace in Multimodal Language Models

• Spatial reasoning remains a substantial challenge, while external answers and actions reveal little about failure sources and internal mechanisms remain poorly understood.Source: §1 · p. 1
• S-Space is a three-dimensional linear workspace whose fixed horizontal, vertical, and distance axes support continuous object-coordinate readouts across datasets, tasks, and prompts.The readouts remain available under non-spatial questions.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Case studies show that S-Space connects visual evidence with linguistic meanings and represents locations beyond directly visible objects.Source: §1 · p. 2
• Viewpoint-change diagnoses find that encoded spatial information can support accurate answers even when models fail to generate them, exposing a gap between perception and computation.Direction-word readouts also reveal coupling between cardinal and image-relative directions.Source: §1 · p. 2
• The paper presents S-Space as a way to characterize how spatial information is encoded, updated, and causally used in MLLM representations.Source: §1 · p. 2

2 Method

• S-Space is a three-dimensional linear subspace of intermediate representations in which object locations are linearly encoded and directly readable.Source: §2 · p. 2

2.1 S-Space Formulation

• For a given model and layer, S-Space is shared across token positions; the search focuses on middle layers between input-anchored early layers and response-oriented later layers.Source: §2.1 · p. 3

2.2 Identifying S-Space

• The method identifies horizontal, vertical, and distance directions by differentiating relative spatial-answer contrasts with respect to object-token activations and aggregating role-antisymmetric gradients.Examples compare query and reference objects along one axis, using logit contrasts such as right versus left.Source: §2.2 · p. 3

Appendix C.1 provides the prompt templates and implementation details.

• The formulation uses image-plane rather than world coordinates for horizontal and vertical axes, without explicitly disentangling depth-dependent perspective effects.The authors state these projected coordinates suffice for most spatial reasoning questions considered.Source: Appendix C.1 · p. 3
• Averaging across templates, object orders, and examples reduces effects that may not cancel in individual cases; the resulting unit vectors span S-Space.The directions capture sensitivity consistently associated with the queried spatial relation.Source: Appendix C.1 · pp. 3–4

2.3 Reading out S-Space

• Once the axes are identified, each object-token representation is projected onto them to obtain coordinates; comparing query and reference coordinates along the queried axis yields a spatial judgment.Positive and negative margins select opposite endpoints such as right versus left.Source: §2.3 · p. 4

3.1 Accuracy

• S-Space coordinates strongly track ground-truth object positions, while coordinate-only spatial QA achieves comparable accuracy to direct answers and generalizes across three benchmarks.Generalization includes CV-Bench, EmbSpatial-Bench, and SpatialTunnel, including benchmarks outside the distribution used to identify S-Space.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Causality

• Coordinate edits substantially alter spatial judgments versus random edits, while unrelated color judgments are largely preserved; common shifts leave relative judgments unchanged.The intervention edits S-Space coordinates and writes them back while leaving the image, question, and representation components outside S-Space unchanged.Source: §3.2 · p. 5

3.3 Persistence

• Across pairwise, single-object, and non-spatial prompts, the same readout supports accurate judgments, suggesting S-Space is not a prompt-template artifact.The readout is evaluated on CV-Bench, EmbSpatial, and SpatialTunnel.Source: §3.3 · pp. 5–6, Figure 5 · p. 6

4 How Does S-Space Emerge? A Case Study

• S-Space is hypothesized to emerge from spatial-language prediction, which may organize stable spatial axes without explicit coordinate supervision.The proposed training pressure is to bind visible entities to relative positions and make that information accessible to language generation.Source: §4 · p. 6
• Readouts place an unseen ball along a goalkeeper’s dive and the observer token near; changing “past” to “toward” shifts the observer closer while the robot stays nearly fixed.Source: §4 · pp. 6–7
• Socialist and conservative project to opposite horizontal ends, mirroring political left/right language and illustrating S-Space’s reach beyond physical space.Source: Figure 7 · p. 7, §4 · p. 7

5.1 Perception-Computation Decomposition via S-Space

• Explicit S-Space readout followed by coordinate transformation yields consistently stronger perspective-taking results, and supplied source-view relations still leave models making errors despite near-perfect S-Space-supported accuracy.This contrast identifies transformation and operation on spatial information as a bottleneck even when the perception premise is provided.Source: §5.1 · p. 8, Figure 8 · p. 8

5.2 Tracking S-Space Transformation through Reasoning

• Across chain-of-thought stages, object coordinates progressively align with the requested viewpoint, but the transformation remains imperfect and does not always yield the correct verbal answer.The advantage shifts from rotated readout earlier to direct readout later in the reasoning sequence.Source: Figure 9 · p. 8, §5.2 · p. 9

5.3 Diagnosing Reference-Frame Coupling in S-Space

• In S-Space, east aligns with right and north with above, suggesting cardinal and image-relative directions use partially shared spatial features.The authors hypothesize this coupling may partly reflect map-like reference frames in web-scale pretraining data.Source: §5.3 · p. 9, Figure 10 · p. 9

6 Discussions

• S-Space supports mechanistic diagnosis of capabilities and missing ingredients, while the discussion suggests dedicated transformation, simulation, and memory tools for spatial reasoning and action.The proposed tool-based approach would avoid requiring one MLLM to perform every spatial operation internally.Source: §6 · pp. 9–10

A Related Work

• S-Space extends Jacobian-based interpretability to continuous spatial coordinates and provides a unified readout, intervention, and viewpoint-tracking interface for real-world images and 3D coordinates.The cited prior work includes spatial representations in synthetic settings and two-dimensional pairwise relations.Source: A Related Work · p. 13

B Theoretical Insights of S-Space

• The proposed analysis assumes a latent three-dimensional subspace whose unit directions represent horizontal, vertical, and distance axes, and decomposes hidden states into spatial, non-spatial, and contextual components.The non-spatial component covers object attributes, while context includes question semantics, prompt phrasing, and scene context.Source: B Theoretical Insights of S-Space · p. 13
• Under Assumptions 1–3, role antisymmetrization recovers the latent spatial direction in expectation.The assumptions characterize gradient decomposition and sign behavior, including cancellation in expectation of role-asymmetric contextual effects.Source: B Theoretical Insights of S-Space · p. 14
• The estimator recovers each direction up to orientation, so the span of the estimated axes matches the latent spatial subspace.The proof states that non-spatial terms vanish and averaging eliminates the remaining contextual difference.Source: B Theoretical Insights of S-Space · p. 14

C.1 Identifying S-Space

• Axis identification and layer selection use disjoint COCO-2017 training sets of 6,000 and 1,800 images, respectively, with 1,000 axis-identification and 300 layer-selection examples per spatial relation.The evaluation includes Molmo2-ER, MolmoAct2-Pretrain, MolmoAct2, Qwen3.5-4B, and Qwen3.6-27B.Source: §C.1 · p. 14
• Horizontal and vertical ground-truth positions come from normalized bounding-box centers, while depth is estimated from inverse depth averaged within each instance mask.Larger inverse-depth values indicate closer objects.Source: §C.1 · p. 14
• For each spatial axis, the procedure uses five prompt templates and exchanges query and reference objects when computing gradients at the object-name-ending token.The listed templates phrase the same left-right, above-below, or far-close judgment in varied wording.Source: §C.1 · pp. 14–15
• Layer selection averages coordinate differences across five templates and chooses the layer with the highest overall accuracy across the three axes.Table 2 reports layer-sweeping accuracy on the COCO validation set, with darker purple indicating higher accuracy and bold marking the selected layer.Source: §C.1 · p. 15, Table 2 · p. 16
• Held-out COCO coordinate correlations use 2,116 pairs per axis; absolute-position correlations include 4,232 object readouts and relative-position correlations use 2,116 coordinate differences.Coordinates are centered by mapping horizontal and vertical values from [0, 1] to [−1, 1] and subtracting the median from depth.Source: §C.1 · p. 16

C.2 Reading out S-Space for Spatial QA

• CV-Bench contributes 1,250 Relation and Depth questions, while EmbSpatial uses a fixed-seed subset of 1,200 questions.The EmbSpatial subset has 200 questions for each of six directions; SpatialTunnel uses all 1,200 evaluation questions, likewise balanced across those directions.Source: §C.2 · p. 16
• Evaluation compares direct generation with S-Space readouts on CV-Bench, EmbSpatial, and SpatialTunnel.Overall accuracy is reported in Table 3, with direction-specific results in Tables 4–6.Source: §C.2 · p. 17, Table 3 · p. 17
• Tables 3–6 organize accuracy as an overall comparison and as direction-specific results across the three benchmarks.Table 3 compares direct answers with coordinate-based S-Space, and Tables 4–6 split accuracy by answer direction; the captions do not provide cell values.Source: §C.2 · p. 17, Table 3 · p. 17, Table 4 · p. 17

C.3 Causal Interventions

• The hidden-state update changes S-Space coordinates to a target while preserving the component orthogonal to S-Space.The update achieves the target coordinates when the three readout axes are linearly independent.Source: §C.3 · p. 18
• Four edits manipulate query and reference coordinates, including signed shifts and swaps whose strength is controlled by α.Shifts preserve the coordinate difference, swaps exchange coordinates at α = 1, and both keep the other two coordinates fixed.Source: §C.3 · p. 18
• Random edits scale query and reference update norms to match the corresponding swap updates at α = 1.The random-edit direction is kept fixed across edit strengths.Source: §C.3 · p. 18
• Tables 7–9 summarize answer retention and layer sweeps for relative-position, absolute-position, and color interventions.Table 8 reports relative-position accuracy against an unedited accuracy of 97.50%; Table 9 reports the ratio of samples whose answer changes under either signed offset.Source: Table 8 · p. 19, Table 7 · p. 19, Table 9 · p. 20

C.4 Case Studies

• Standardized horizontal readouts preserve the ordering of socialist, moderate, and conservative across all five models.Figure 12 includes two MolmoAct2 models alongside the three models shown in Figure 7.Source: §C.4 · p. 20, Figure 12 · p. 22
• Direction-word readouts project generated direction terms onto COCO-identified horizontal and vertical axes, aggregating 1,311 occurrences across questions.Compound directions written as one word, hyphenated, or spaced count as a single occurrence.Source: §C.4 · p. 20
• Figure 11 presents S-Space readouts for unseen entities, the observer, and moving entities such as a robot.The readouts include horizontal or distance coordinates and report means with 95% bootstrap intervals across prompt templates.Source: §C.4 · p. 20, Figure 11 · p. 21

C.5 Perspective-Taking Spatial Reasoning

• SpinBench tests leftward appearance or closeness from left, right, and back views, with direct generation, reasoning, and S-Space + Rotation compared across five models.The evaluation uses 146 questions without a premise and 144 with one; controlled CoT supplies the original relation, target viewpoint, and rotated relation using benchmark labels.Source: §C.5 · p. 22, Table 10 · p. 23
• The viewpoint computation rotates horizontal and depth coordinates, then selects the object meeting the requested leftward or closer criterion.With additional spatial premises, Molmo2-ER's S-Space coordinates become more accurate and the readout-and-rotation pipeline answers perspective-taking questions more reliably.Source: §C.5 · pp. 22–23, Figure 13 · p. 23

C.6 S-Space under Action Training

• The analysis asks whether learning to act sharpens S-Space geometry because reaching and manipulation require resolving scene geometry and functional parts.It compares the VLM with its fine-tuned VLA models.Source: §C.6 · p. 23
• The comparison uses Molmo2-ER, MolmoAct2-Pretrain, and MolmoAct2 on InstructPart images with estimated depths for each object and its functional part.Ground-truth depth comparisons use the provided object and part masks with Depth Anything V2-Small.Source: §C.6 · p. 23
• Depth geometry between a manipulable object and its functional part becomes increasingly faithful to their physical relation from VLM to action-model pretrain to VLA.Figure 14 compares kettle-handle depth readouts and depth-comparison accuracy at L18 for all three models.Source: §C.6 · p. 23, Figure 14 · p. 24

C.7 S-Space in Multi-View Contexts

• Because S-Space is identified from single-view examples, the multi-view test asks whether language can select which view defines the observer-centered frame.Source: §C.7 · p. 24
• The test uses 600 panoramic-scene view pairs, with 100 per direction, keeping the “In View 1” query fixed while swapping which image is designated View 1.Each pair contains the same object in both views.Source: §C.7 · p. 24
• Steering accuracy is consistently well above random choice across the horizontal, vertical, and distance axes, suggesting S-Space can be dynamically anchored to the viewpoint selected through language.The prompt-steering test evaluates whether coordinate changes follow an object's ground-truth displacement between views.Source: Figure 15 · p. 24, §C.7 · p. 24