PaperlayerView exact revision on arXiv

Source-linked AI summary

Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?

Pingyue Zhang, Zihan Huang, Yue Wang, Jieyu Zhang, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Manling Li

arXiv:2602.07055v1cs.AIcs.CLcs.LG

TL;DR

Spatial embodied intelligence requires agents to gather information and maintain spatial beliefs under partial observability, but active exploration by foundation models remains insufficiently evaluated. THEORY OF SPACE addresses this gap with active exploration, downstream spatial tasks, and explicit cognitive-map probing, finding degraded active performance, inefficient exploration, unstable beliefs, and persistent obsolete priors.

  • Problem

    Active, self-directed spatial exploration and belief construction by multimodal foundation models remain understudied despite their importance under partial observability.

  • Method

    THEORY OF SPACE evaluates agents in text- and vision-based partially observable worlds through active exploration, route and survey tasks, and explicit cognitive-map probing.

  • Results

    Foundation models show an active-passive performance gap, inefficient exploration, unstable spatial beliefs, and belief inertia when environmental changes make prior coordinates obsolete.

  • Takeaways & Limitations

    Current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.

  • Takeaways & Limitations

    Future work must extend THEORY OF SPACE beyond single-agent settings to address coordination and spatial-belief alignment or sharing across agents.

Abstract

from arXiv · show

Spatial embodied intelligence requires agents to act to acquire information under partial observability. While multimodal foundation models excel at passive perception, their capacity for active, self-directed exploration remains understudied. We propose Theory of Space, defined as an agent's ability to actively acquire information through self-directed, active exploration and to construct, revise, and exploit a spatial belief from sequential, partial observations. We evaluate this through a benchmark where the goal is curiosity-driven exploration to build an accurate cognitive map. A key innovation is spatial belief probing, which prompts models to reveal their internal spatial representations at each step. Our evaluation of state-of-the-art models reveals several critical bottlenecks. First, we identify an Active-Passive Gap, where performance drops significantly when agents must autonomously gather information. Second, we find high inefficiency, as models explore unsystematically compared to program-based proxies. Through belief probing, we diagnose that while perception is an initial bottleneck, global beliefs suffer from instability that causes spatial knowledge to degrade over time. Finally, using a false belief paradigm, we uncover Belief Inertia, where agents fail to update obsolete priors with new evidence. This issue is present in text-based agents but is particularly severe in vision-based models. Our findings suggest that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.

1 INTRODUCTION

THEORY OF SPACE frames spatial intelligence as active exploration under partial observability, requiring agents to construct and use internal spatial beliefs. Its benchmark exposes substantial gaps in autonomous information gathering, exploration efficiency, and belief revision.

  • THEORY OF SPACE treats exploration as a first-class decision problem for constructing, revising, and exploiting internal spatial beliefs.The framework also probes uncertainty, exploration steps, and information resolved per action.
  • The benchmark requires agents to move, rotate, and observe in text- and vision-based worlds before evaluating route-level and survey-level spatial knowledge.Scripted proxy agents provide strong reference trajectories for separating exploration from reasoning.
  • 57.1 →46.0 for GPT-5.2 and 60.5 →57.3 for GEMINI-3 PRO show the active-passive performance gap in vision-world evaluation.The comparison is between passive reasoning and active exploration settings.
  • ∼9 steps versus ≥14 steps shows that rule-based proxies reach target coverage sooner than foundation models without improving belief accuracy.Foundation models explore redundantly and fail to autonomously structure information gathering.
  • False-belief evaluations reveal spatial belief inertia: agents, especially vision-based ones, retain obsolete coordinates after directly observing changed object configurations.This identifies a failure to revise spatial memory in response to environmental changes.
  • Extending THEORY OF SPACE to multi-agent exploration remains future work because coordination and sharing or aligning spatial beliefs introduce additional challenges.

2 THEORY OF SPACE

THEORY OF SPACE defines spatial intelligence as constructing, revising, and exploiting a probabilistic spatial belief from partial observations. The evaluation combines active exploration, downstream route and survey tasks, and explicit cognitive-map probing.

  • THEORY OF SPACE is the ability to construct, revise, and exploit an internal spatial belief.The concept models uncertain, currently unobserved spatial structure rather than hidden mental states of other agents.
  • An internal spatial belief is a working-memory mental model of spatial layout and relations updated from partial observations.The formalism represents the belief as Bt over spatial structure given an observation-action history.
  • Construct integrates partial observations into a globally consistent belief approximating the posterior P(S | ht).
  • Revise updates prior beliefs with new information after environmental shifts, while exploit conditions downstream task actions on the current belief.Exploitation is evaluated by downstream task performance under a belief-conditioned policy.
  • Active exploration requires agents to select moving, rotating, observing, and terminating actions in partially observable environments rather than merely answer downstream questions.The evaluation focuses on deciding what to see next.
  • The paradigm evaluates route knowledge, survey knowledge, and cognitive-map correctness, coherence, temporal integration, and uncertainty about unobserved regions.Explicit probing makes the internal spatial representation directly measurable rather than inferred only from behavioral success.

3 BENCHMARKING THEORY OF SPACE ABILITY FOR FOUNDATION MODELS

The benchmark separates active spatial-belief construction from downstream reasoning, using procedurally generated environments, multimodal observations, and route- and survey-level tasks. It evaluates exploration efficiency, belief quality and revision, uncertainty modeling, and exploitation success across active and passive settings.

  • Benchmark structure: The benchmark has an exploration phase for constructing a spatial belief and a reasoning phase for performing downstream spatial tasks.Agents select and execute actions to gather local observations before answering spatial questions.
  • Spatial environment construction: Procedurally generated multi-room layouts use an N × M grid, with objects assigned 2D coordinates and cardinal orientations.Each random seed deterministically instantiates a distinct scene, supporting scalable and controlled experiments.
  • Spatial environment construction: Agents interact through Goto, Rotate, Observe, and Query actions, with Observe and Query assigned costs of 1 and 2.The action space targets high-level information gathering rather than low-level motor control.
  • Downstream spatial tasks: Route belief tests egocentric path and landmark relations, whereas Survey belief tests allocentric mapping, geometric transformation, and perspective conversion.Together, the tasks assess local navigation reasoning and global spatial abstraction.
  • Assessment dimensions: Assessment dimensions separate belief construction and revision from exploitation, measuring exploration efficiency, cognitive-map quality, uncertainty modeling, belief revision, and task success.Construction and revision are evaluated through efficiency and belief quality, while exploitation is evaluated through downstream task success.
  • Evaluation settings: The benchmark distinguishes acquiring information from synthesizing it by comparing active on-policy exploration with passive reasoning over logs generated by scripted proxies.Active agents plan actions to reduce uncertainty, while passive agents reason over standardized proxy trajectories.

4 EVALUATION AND ANALYSIS

The evaluation compares autonomous active exploration with passive observation across text and vision environments, measuring spatial-task performance, exploration efficiency, and scaling across room configurations. Models show an active–passive gap, incomplete coverage, modality differences, and inefficient or unsystematic exploration.

  • Active agents autonomously plan exploration and are evaluated on exploration cost, route-level reasoning, and survey-level reasoning across text and vision environments.
  • Passive evaluation decouples spatial reasoning from exploration by using standardized observation logs generated by scripted proxy explorers.
  • Text performance significantly exceeds vision performance, while humans outperform foundation models in both domains, especially in vision.
  • ≈9 steps: the SCOUT proxy reaches target coverage, whereas autonomous models use significantly more actions without performance benefit; guided text trajectories yield 83.9 for GPT-5.2 and 86.7 for GEMINI-3 PRO.
  • Active accuracies underperform passive performance because incomplete exploration leaves uncertainty and models often terminate prematurely.
  • As room number increases, exploration cost rises, performance declines, and the active–passive gap widens for GPT-5.2 and GEMINI-3 PRO.
  • GPT-5.2 explores unsystematically through door-first behavior, object omission, and path redundancy, whereas GEMINI-3 PRO uses a more methodical rotate-and-scan strategy.

5 HOW DO FOUNDATION MODELS MANAGE INTERNAL SPATIAL BELIEF?

Belief probing exposes how models construct and maintain internal spatial maps from sequential observations. The results show modality-specific perception failures, temporal instability, and persistent revision errors, while validating the cognitive map as a useful diagnostic.

  • Belief probing: Agents externalize evolving spatial beliefs as structured cognitive maps containing global layouts and local observations.The global map is anchored to the initial pose, while the local map records currently visible objects and pose.
  • Cognitive map failures: Vision models show a substantial modality gap, with especially weak object-facing recognition and near-chance orientation correctness.The text reports that vision performance drops across metrics and connects orientation weakness to poor perspective-taking accuracy.
  • Cognitive map failures: Correct local perception often degrades into incorrect final maps because previously verified positions and orientations are overwritten over time.This instability produces belief drift and lowers final map correctness.
  • Validation: ≈95% performance with oracle cognitive maps shows that the map representation contains the information needed for evaluation tasks.The result attributes the main bottleneck to constructing accurate maps rather than to the representation format.
  • Validation: Explicitly generating a cognitive map slightly reduces performance, indicating an externalization gap between latent beliefs and lossy JSON maps.Despite this gap, map correctness remains positively correlated with downstream success, with stronger reported correlations in vision than text.
  • Belief revision: Vision agents are less accurate and more redundant when revising beliefs after object changes, especially for altered orientations.They frequently retain obsolete facing estimates despite new visual evidence.

6 RELATED WORK

Prior spatial-reasoning benchmarks primarily test passive inference or active exploration tied to particular downstream tasks. This work targets general spatial-belief construction through task-agnostic exploration.

  • Passive Spatial Reasoning: Passive benchmarks ask agents to answer spatial queries from textual descriptions, single images, or multiple views.These settings provide observations without requiring agents to choose actions that gather information.
  • Active Exploration for Spatial Understanding: Active-exploration benchmarks typically evaluate evidence gathering for embodied question answering or long-horizon instruction-following tasks.Their exploration objectives are generally coupled to task-specific goals rather than construction of a general spatial belief.

7 CONCLUSIONS

THEORY OF SPACE evaluates whether foundation models can actively acquire information under partial observability to construct, revise, and exploit spatial beliefs. Its experiments show that active exploration, belief maintenance, and revision remain major weaknesses, especially in vision settings.

  • Conclusion: THEORY OF SPACE frames spatial agency as self-directed information acquisition for constructing, revising, and exploiting internal spatial beliefs.The benchmark evaluates this process rather than reasoning only from fixed views.
  • Conclusion: Active exploration reduces end-task performance and is less efficient than passive viewing, with the gap widening as room complexity increases.Belief probes identify early visual perception errors and later instability that overwrites or forgets correct information.
  • Conclusion: When environments change, models struggle to revise obsolete spatial beliefs, revealing difficulties with coherent and revisable spatial memory.The conclusion links these failures to the broader challenge of maintaining spatial beliefs during active exploration.

A.1 BENCHMARK CONSTRUCTION

The benchmark uses procedurally generated text and vision worlds with discrete actions, structured observations, uncertainty propagation, and scripted exploration proxies. Prompt designs and parallel representations support controlled evaluation of active spatial belief construction.

  • Benchmark environment: The environment exposes a Gym-like, partially observable interface over procedurally generated multi-room scenes arranged on grid layouts.Objects have integer coordinates and cardinal headings, while agent spawn poses are randomized.
  • Observation modalities: Text worlds provide symbolic object observations within a 90° forward field of view, including discretized direction, distance, identity, and facing cues.Visibility is bounded by rooms and doorways.
  • Observation modalities: Vision worlds use 3D scenes containing 293 household-object models across 6 categories and 37 subtypes.Each object type appears at most once per scene, with placement constraints and reproducible random seeds.
  • Prompt design: Distance and orientation reference images calibrate visual observations for discretized distance bins and object-facing recognition.The prompts show colored distance cues and front or side views depending on whether objects have inherent orientation.
  • Information gain: AC-3 maintains feasible grid-cell domains for objects and propagates observation-derived constraints until the domains reach a fixed point.Because constraints come from valid trajectories, the ground-truth assignment remains supported and domains stay non-empty.
  • Proxy agents: SCOUT and STRATEGIST provide reproducible exploration baselines, using room sweeps or uncertainty-guided viewpoints respectively.SCOUT follows visit–sweep–advance, whereas STRATEGIST selects objects with the largest remaining positional domains.
  • Prompt design: Separate prompts probe exploration, downstream evaluation, cognitive-map externalization, and uncertainty over candidate points in parallel text and vision worlds.The prompt figures document these task and belief-probing interfaces.

B EVALUATION SETUPS

The evaluation compares text and vision settings using reproducible room layouts, then reports passive and active exploitation performance across two-, three-, and four-room configurations.

  • Evaluation Design: 100 evaluation instances use IDs 0–99 as shared random seeds across text and vision environments, guaranteeing deterministic layouts and bit-for-bit reproducibility.The same seed drives task sampling in both modalities for each instance.
  • Room Settings: Two-room and four-room layouts retain the three-room setting’s room size and objects-per-room configuration.In the four-room setting, the main room connects to all other rooms.
  • Reported Results: Tables 8 and 9 report exploitation performance under passive observations and active exploration for two-room layouts.The corresponding four-room passive and active results are reported in Tables 10 and 11.

C ADDITIONAL VISUALIZATION EXAMPLES

The additional visualizations document task formats, cognitive maps, exploration trajectories, and analysis-platform views across text and vision environments. Together, they illustrate model-specific exploration behaviors and the stronger cognitive-map performance observed in text-based environments.

  • Cognitive Maps: Figures 15 and 16 visualize GPT-5.2’s turn-by-turn cognitive maps in text and vision worlds against ground-truth maps.The accompanying comparison reports noticeably stronger performance in text-based environments than in vision-based ones.
  • Task Formats: Figure 14 presents Route-level and Survey-level spatial reasoning tasks with open-ended answers constrained to strict output formats.In vision settings, rendered observation images replace textual scene descriptions in the questions.
  • Exploration Patterns: GPT-5.2’s representative trajectories include a door-finding strategy that rotates after detecting a door to maximize environmental coverage.The process terminates after all target objects are identified.
  • Exploration Patterns: GPT-5.2 may omit further exploration when it observes a door too early, producing incomplete environmental discovery.This trajectory illustrates an omission pattern distinct from its systematic sweeping behavior.
  • Exploration Patterns: GEMINI-3 PRO uses door-directed systematic sweeping in vision worlds and pivot-based object sweeping with random door transitions in text worlds.The text-world pattern repeatedly orbits the current object after jumping to a new observed door.
  • Exploration Patterns: CLAUDE-4.5 SONNET’s example trajectory has no clear exploration pattern.The visualization is presented as a representative model-specific exploration example.
  • Analysis Platform: Figures 22–26 demonstrate an analysis platform covering active-exploration metrics and individual turns with actions and cognitive maps in text and vision worlds.The platform includes separate visualizations for text-world and vision-world analysis.
Loading 2602.07055v1…