PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing Spatial State Representations in Visual Foundation Models

Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu

arXiv:2610.40636v1cs.CVcs.AI

TL;DR

Visual foundation models may encode visible scenes without maintaining spatial state beyond the current view, updating it under self-motion, or using it for action. The paper probes four spatial capabilities across ten frozen representations and finds that results vary across tasks, layers, and prefixes, with decodability, input use, and goal-directed utility dissociating.

  • Problem

    Whether visual foundation models support spatial state beyond visible-scene readout remains an open question, since success on a training objective or one spatial level does not establish the others.

    Source: §1 · pp. 2–3

  • Method

    The authors probe ten frozen representations across four spatial capabilities and use matched controls to test reliance on history, motion, and goals.

    Source: §1 · p. 3

  • Results

    Across tasks, representations, layers, and prefixes, readout accuracy, input use, and goal-specific utility dissociate, with persistence and actionability linked to camera-pose readout and feature prediction to visible-scene readout.

    Source: §1 · p. 3, §6 · p. 13

  • Takeaways & Limitations

    Evaluating spatial representations requires testing whether information persists beyond observation, updates predictively, and supports goal-directed action, not only whether it can be decoded.

    Source: §1 · p. 3, §6 · p. 13

    Some validation scenes overlap Fast3R's ScanNet++ training data, so its leading camera-pose, persistence, and action results may partly reflect scene familiarity.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.

1 Introduction

The paper frames spatial perception as maintaining and updating an observer-relative state beyond the visible scene, then tests whether visual foundation models support that state. Its four-level probing study finds that capabilities vary across tasks, layers, and prefix lengths, and that readout, input use, and goal-directed utility can dissociate.

Motivation

  • The paper motivates spatial state as more than image recognition: observers maintain information about unseen objects, update location during movement, and use this information to guide behavior.These abilities are described as arising from perception and sensorimotor interaction, not inherently from language.Source: §1 · p. 1

Research framing

  • The study operationalizes spatial state through four capabilities: visible-scene structure, persistence of out-of-view objects, predictive updating under ego-motion, and goal-conditioned action.It does not prescribe a particular map, scene graph, coordinate system, or neural architecture.Source: §1 · p. 2, Figure 1 · p. 2
  • The four levels demand progressively more than the current observation: reading the visible scene, retaining earlier views, anticipating unobserved views, and relating state to a goal and action.Which lower-level ingredients higher levels use is treated as an empirical question.Source: §1 · p. 2, Figure 2 · p. 3

Study overview

  • The authors probe ten frozen representations across network depth and streaming context, using controls that vary motion, history, transformation order, or goals while holding predictors and targets fixed.The study focuses on observer motion through static indoor environments, excluding object dynamics, contact physics, and complete physical world modeling.Source: §1 · p. 3

Findings

  • Across ten representations, no model leads on every task, and capability profiles vary with layer and prefix length; persistence and actionability build on camera-pose readout, while feature prediction builds on visible-scene readout.Matched controls further separate readout accuracy from reliance on history and motion, and goal changes affect predicted actions without reliably improving progress.Source: §1 · p. 3

2 Probing Spatial State Representations

The probing framework asks what spatial information frozen visual features expose, what inputs support it, and whether it remains useful for prediction and goal-directed action. It instantiates four capability levels with task-specific probe inputs and targets, using independently encoded causal prefixes rather than an online recurrent state.

2.1 Four functional capabilities

  • L1 measures visible-scene structure through camera pose and depth, object-instance identity, and feature correspondence across viewpoints.These tests assess metric geometry, object-level organization, and view consistency, respectively.Source: §2.1 · p. 4, Figure 3 · p. 4
  • L2 tests whether a probe can locate a previously visible object relative to the final camera after that object leaves view, given prefix features and an object query.The target is the object's bearing; visibility masks and target locations are not supplied to the probe.Source: §2.1 · p. 4, Figure 3 · p. 4
  • L3 tests direct future-feature prediction and sequential path integration from relative camera motions; their different probe architectures mean the comparison does not isolate recurrence alone.The main direct-prediction targets use offsets h ∈ {1, 2, 4}; MiniMax-H3 uses h = 1.Source: §2.1 · p. 4
  • L4 tests goal-conditioned action prediction using the current visual history and an independently encoded goal frame to predict movement toward that goal.The task asks whether descriptive or predictive spatial information can also support behavior selection.Source: §2.1 · pp. 4–5, Figure 3 · p. 4

2.2 Causal streaming formulation

  • Each causal prefix is encoded independently and contains no frame after its endpoint; longer-prefix comparisons also move the endpoint and therefore do not isolate added history.For L2, the object query comes from frames where the object was visible, while its final-camera bearing is withheld; L3 and L4 use an independently encoded future frame.Source: §2.2 · p. 5

3 Experimental Setup

Experiments use frozen representations on posed ScanNet++ trajectories, with scene-level partitions and task-specific protocols. The main comparisons average designated-layer scores across evaluated prefixes, while layer and prefix sweeps and uncertainty estimates are reported separately.

3.1 Data and splits

  • Fast3R's training data include ScanNet++, and some validation scenes overlap; its leading camera-pose, object-persistence, and action results may partly reflect familiarity with those scenes.The temporal experiments use posed ScanNet++ trajectories with RGB observations, camera poses, instance geometry, and visibility annotations.Source: §3.1 · p. 5

3.2 Models and extraction points

  • The study compares ten frozen representations spanning image and video encoders, 3D and multimodal models, video generators, and world models.Layer indices are model-specific, not absolute measures of depth across architectures.Source: §3.2 · p. 5, Table 1 · p. 5
  • Main scores average evaluated prefixes at designated layers, with layer and prefix sweeps analyzed separately rather than using post-hoc maxima as primary scores.The shared readout architecture and objective are held constant across layers and prefix lengths within each model and task.Source: §3.2 · p. 5
  • The main table reports final-checkpoint evaluations on 276 validation windows per prefix, with task-specific window and candidate counts; uncertainty intervals are scene-clustered bootstrap intervals over 69 scenes.Those 95% intervals reflect evaluation sampling, not training-seed variability.Source: §3.2 · p. 6, Table 2 · p. 6

4 Experiments

Across ten representations, spatial readouts vary by task, layer, and prefix, and success on one capability does not guarantee success on another. The experiments test what probes read from history and motion, whether spatial structure remains readable after prediction, and whether goal-conditioned actions improve progress.

4.1 Spatial capabilities differ across representations

  • Fast3R leads camera-pose readout and object persistence, DINOv2 leads direct prediction, and V-JEPA2 leads path integration; persistence and prediction produce different model rankings.Fast3R’s persistence error is 32.79°, while DINOv2’s direct-prediction R@1 is 0.2373 and V-JEPA2’s path-integration score is 0.1588.Source: §4.1 · pp. 6–7

4.2 Spatial accessibility depends on layer and streaming prefix

  • Spatial readouts favor different layers, and longer prefixes often worsen them: 45 of 56 endpoint pairs favor p8, including all eight object-persistence pairs.The extraction setting is consequential: the longer prefix also moves the endpoint, so the comparison does not isolate history alone.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 What do the probes actually use?

  • Matched controls show correct motion improves candidate retrieval for V-JEPA2, VideoMAE, and CogVideoX, while full history improves localization for V-JEPA2 and Wan, but not measurably for VideoMAE.VideoMAE’s strong localization score therefore does not establish that its probe uses accumulated visual history.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Prediction changes the spatial readout interface

  • V-JEPA2’s native predicted features retain some recoverable spatial structure, but a reader trained on real features loses accuracy when transferred unchanged.At horizon two, normalized-depth MAE rises from 0.182 on real features to 0.246 on native predictions; retraining improves it to 0.223.Source: §4.4 · p. 10, Figure 5 · p. 10

4.5 Goal-conditioned action prediction

  • Candidate selection uses goal information, but continuous-output actions respond to goal changes without reliably improving progress when given the correct goal.FULL exceeds GOAL-ONLY by 6.0–13.8 percentage points but does not improve positive progress over CURRENT-ONLY for any backbone; these are offline diagnostics, not a closed-loop policy.Source: §4.5 · pp. 10–11, Figure 5 · p. 10, Table 4 · p. 11

4.6 How the four levels relate

  • Across capability levels, object persistence and actionability track camera-pose readout, whereas feature prediction tracks visible-scene readout; lower-level readout bounds but does not guarantee higher-level performance.Fast3R leads both action settings yet ranks eighth on direct prediction, while DINOv2 leads direct prediction.Source: §4.6 · pp. 11–12

5 Related Work

Related work spans frozen-feature geometry probes, language-based spatial evaluations, object-centric persistence, and embodied world models. This paper extends these lines by testing multiple functional levels of spatial state in the same frozen representations under matched controls.

Probing spatial and 3D structure in frozen representations

  • Frozen-feature geometry studies probe spatial signals across layers, while reconstruction and streaming-memory models provide geometry-specialized or persistent-state comparisons.The paper treats metric geometry as one diagnostic among several and does not assume an explicit state in general-purpose representations.Source: §5 · p. 12

Internal representations of diffusion and video generation models

  • Diffusion and video-generation probes motivate analyzing selected layers, generation coordinates, and streaming prefixes rather than treating one hidden state as representative.These profiles describe information accessibility during computation, not when a capability emerges during training.Source: §5 · p. 12

Spatial representations in foundation models

  • Language-based spatial benchmarks and representation probes assess different abilities; this study reads spatial structure, persistence, predictive update, and goal use from the same ten representations.Prior probes report variation in 3D structure, object positions, camera motion, and motion direction across models and network depth.Source: §5 · p. 12

World models and embodied representations

  • World models use predicted observations or features for interaction and planning, whereas these L3 and L4 probes ask what ego-motion updates and goal-conditioned actions are readable from frozen features.The cited work includes action-controllable environments, feature-space planning, and goal-conditioned visual planning.Source: §5 · p. 13

Object-centric representations and persistence

  • Object-centric systems propagate entity-level slots, but this study instead tests whether bearing remains recoverable after an object leaves view and which visual history supports that readout.This operational persistence test differs from benchmarks that discriminate possible from impossible events.Source: §5 · p. 13

6 Discussion and Conclusion

The discussion characterizes spatial state through four functional levels and finds that capabilities differ across models, layers, and prefixes. Its conclusions are bounded to frozen-feature interfaces, trained probes, and offline indoor trajectories.

6 Discussion and Conclusion

  • Camera-pose readout supports object persistence, which in turn supports goal-conditioned action, while feature prediction builds on visible-scene readout and is unrelated to persistence.The paper reports these as two distinct ingredient chains across the four functional levels.Source: §6 · p. 13
  • Matched controls show that decodability, input use, and functional utility can dissociate across persistence, predicted features, and goal-sensitive action readouts.A natively predicted feature retains recoverable structure, but a reader trained on real features does not transfer unchanged; a goal-sensitive action readout need not benefit from the correct goal.Source: §6 · p. 13
  • The conclusions concern independently encoded causal prefixes and offline indoor trajectories, not incremental state updates or a shared physical error scale for retrieval scores.The study evaluates frozen feature interfaces and trained probes.Source: §6 · p. 13

A.1 Additional layer, history, and intervention analyses

Additional analyses examine extraction-depth profiles and test which historical visual evidence supports object-persistence readout.

A.1 Additional layer, history, and intervention analyses

  • The main text keeps compact summaries, while full measured layer profiles, joint instance-identity and localization grids, and an intervention schematic are provided separately; layer scans describe extraction-depth profiles, not training-time emergence.Source: §A.1 · p. 17
  • Removing target-visible frames or zeroing visual tokens increases localization error for V-JEPA2 and Wan, while VideoMAE and CogVideoX show no reliable change.The object query remains available in every intervention.Source: §A.1 · p. 17
  • Shuffling frame contents changes error by at most 0.95° with intervals containing zero, and reversing history raises V-JEPA2 error by 2.15° [0.22, 4.56] while other models change by less than 2° with intervals containing zero.Because shuffling and reversal preserve the final frame and temporal identifiers, these tests concern historical content rather than ordered state updating.Source: §A.1 · p. 17

A.2 Comparison on shared validation windows

The appendix checks whether spatial-readout comparisons persist on shared validation windows and under a matched prediction horizon. It reports stable qualitative model orderings while noting that the next-observation advantage of two image encoders may partly reflect feature continuity.

A.2 Comparison on shared validation windows

  • On 171 shared validation windows, most model orderings remain stable, with only instance-identity and lower-ranked persistence ordering swaps; path integration keeps V-JEPA2 first.The shared-window comparison aligns validation windows, not training sets or prediction protocols, and scores are compared within each table because window sets and gallery sizes change.Source: §A.2 · p. 17, Table 6 · p. 18
  • With the horizon and gallery matched to 171 next-observation candidates, model ordering is essentially unchanged (ρ = 0.96), while DINOv2 and Qwen2.5-VL reach next-observation R@1 of 0.61 and 0.53.No predictor is retrained; MiniMax-H3 was trained on h = 1, while the other models were trained on h ∈ {1, 2, 4}.Source: §A.2 · p. 17
  • Figure 6 plots direction-corrected percentile-rank profiles across normalized backbone depth, with isolated output representations in a separate right-hand band.Metric geometry averages depth and camera-pose ranks; hollow MiniMax-H3 markers use the 171-window subset, with h = 1 only for direct prediction.Source: Figure 6 · p. 18
  • The DINOv2 and Qwen2.5-VL next-observation advantage may partly reflect feature continuity between adjacent frames rather than motion-conditioned prediction.A copy baseline would separate these possibilities for these models.Source: §A.2 · pp. 17–18

A.3 Scene-bootstrap intervals

The reported intervals use paired scene resampling to compare evaluation cells, but they do not capture variability across training seeds.

A.3 Scene-bootstrap intervals

  • Intervals resample the 69 validation scenes 1,000 times with one shared draw matrix, pairing model, prefix, and coordinate comparisons.Metrics are recomputed from saved per-sample predictions as ratios of per-scene sums; point estimates match the reported values.Source: §A.3 · p. 18
  • For retrieval, the gallery remains the full validation set while queries are resampled; action gains are paired per query.Source: §A.3 · p. 18, Table 7 · p. 20
  • Table 8 reports paired 95% scene-bootstrap intervals for the same designated-layer, prefix-averaged cells as Table 2.The best complete result in each column and results statistically indistinguishable from it are bolded.Source: Table 8 · p. 20
  • The intervals reflect evaluation sampling for a single training seed, not training-seed variability.Source: §A.3 · p. 20

A.4 Generation coordinate analysis

Generation-coordinate effects vary by model and readout, although object persistence favors the earlier coordinate in all three models.

A.4 Generation coordinate analysis

  • The coordinate study evaluates layers 15, 20, and 25 and prefixes 8, 12, 16, and 24, using model-specific scheduler or flow coordinates.The compared coordinates are not interpreted as a shared physical noise level.Source: §A.4 · p. 20
  • Wan favors the lower coordinate on depth, view consistency, and persistence in every pair; CogVideoX does so on persistence in all twelve pairs.CogVideoX favors the lower coordinate on depth in nine of twelve pairs and view consistency in ten of twelve.Source: §A.4 · pp. 20, 22
  • Object persistence favors the lower scheduler coordinate for Wan, CogVideoX, and MiniMax-H3 across layer–prefix pairs.Figure 9 shows layer–prefix patterns; Table 12 reports block-20 means over four prefixes.Source: Figure 9 · p. 22, §A.4 · p. 22, Table 12 · p. 22
  • MiniMax-H3 favors the higher coordinate on depth in ten of twelve pairs and instance identity in nine, but the lower coordinate on view consistency and persistence.Instance-identity and view-consistency differences at block 20 are within paired intervals; the table marks differences whose intervals exclude zero.Source: Table 11 · p. 21, §A.4 · p. 22, Table 12 · p. 22

A.5 Motion composition consistency

Across these appendices, the probes test motion composition, sensitivity to supplied motion, predictive-readout controls, and goal-conditioned action readouts under distinct protocols.

A.5 Motion composition consistency

  • All evaluated Wan and V-JEPA2 layers improve endpoint prediction over copying the initial state; V-JEPA2 block 11 has a smaller normalized composition distance than Wan block 15 (0.0232 versus 0.0377).The external recurrent updater and readout are trained only on one-step camera transitions; the comparison uses predeclared representative layers.Source: §A.5 · p. 23
  • Action-only, matched-shuffle, and additive controls reveal no stable order effect, limiting attribution of weak latent order effects to the representation.The analytic SE(3) oracle has a positive order margin, while a geometric GRU has no reliable order margin under the same one-step objective.Source: §A.5 · p. 23

A.6 Action sensitivity of direct future prediction

  • 0.029 and 0.006 are the clearly positive cosine margins for DINOv2 and V-JEPA2; matching accuracy exceeds chance across models, but motion sensitivity remains small.MiniMax-H3’s h=1-trained head reaches 40.74% within-set accuracy at block 30 when tested on h=2 and h=4 motions.Source: §A.6 · pp. 23–24, Table 14 · p. 24

A.7 Additional protocol details

  • The one-step and history-intervention studies use matched sample counts and paired scene-clustered intervals; localization uses a historical object mask without supplying that mask or camera pose to the readout.Content shuffling and reversal retain the final frame and temporal identifiers; moving tokens with their identifiers is only a permutation sanity check.Source: §A.7 · p. 24

A.8 Supporting controls for predictive spatial readout

  • Real-target depth readout beats the layout prior, while true-motion external prediction lowers depth error versus identity, shuffled-motion, and context-only controls.For Wan, the reductions are 0.003–0.007 at both horizons; for V-JEPA2, true motion reduces MAE versus context-only by 0.0081 at h=2 and 0.0064 at h=4.Source: §A.8 · p. 24

A.9 Action readout protocol and controls

  • In the continuous-output setting, FULL has 2.0–3.6 percentage-point lower positive progress than CURRENT-ONLY; only V-JEPA2’s interval excludes zero.Mean goal reduction is also lower for all four representations, with intervals excluding zero for DINOv2, V-JEPA2, and Wan; candidate-selection and continuous-output evaluations remain distinct.Source: §A.9 · p. 25, Table 16 · p. 25

A.10 Predictive readout and action prediction metrics; A.11 Feature extraction details

  • Tables 15 and 16 provide numerical readouts and action metrics alongside Figure 5, while Table 17 lists probed layers and extraction coordinates.Generative-model features use specified denoising or flow coordinates; the separate coordinate study also evaluates blocks 15 and 25.Source: §A.10 · p. 25, §A.11 · p. 26, Table 17 · p. 26

Related papers

Loading 2610.40636v1…