PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialBrain: Spatial Intelligence Growing from Experience

Yifan Zhou, Ziyang Gong, Xiaojun Ye, Yifei Liu, Donggang Li, Xiaolong Zhou, Zhihang Zhong, Xue Yang

arXiv:2610.29317v1cs.AI

TL;DR

Spatial reasoning requires selecting the right computation because tasks may need direct observation, geometric evidence, or reference frames, and reusable tools alone do not determine when to use them. SpatialBrain combines current-scene memory, reusable spatial skills, and accumulated experience about resource use while keeping model weights fixed. It improves over direct answers on four of five benchmarks and ties on the fifth.

  • Problem

    Spatial tasks vary in whether they require direct observation, reference frames, multiple-view geometry, or route tracking, and available resources alone do not specify when they are useful.

    Source: §1 · p. 1

  • Method

    SpatialBrain combines queryable current-scene evidence, reusable parameterized spatial procedures, and accumulated experience about when and how to use resources, with fixed model weights.

    Source: §1 · p. 2

  • Results

    SpatialBrain improves over direct answers on four of five benchmarks and ties on the fifth.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework distinguishes what the current scene supports from reusable procedures and experience guiding resource selection.

    Source: §1 · p. 2

    A skill may query current-scene memory but cannot supply an earlier answer or bypass a disabled memory interface.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Spatial reasoning requires more than access to geometric tools: a reasoner must decide when direct observation is sufficient and when to reconstruct, measure, or change reference frames. We present SpatialBrain, a framework that retains experience about these decisions while keeping its vision-language backbone fixed. The key distinction is between evidence about a scene and experience about how to reason with that evidence. 3DFusionMemory fuses current-scene observations into queryable spatial relations. A Spatial Skill Library provides reusable tool procedures, and Spatial Cognition associates task conditions with experience about when and how to use memory and skills. Completed trajectories are abstracted into process records and consolidated between task blocks, allowing later decisions to draw on earlier execution without transferring scene coordinates or reference answers. Across five spatial-reasoning benchmarks, SpatialBrain improves on direct answers from the same backbone on four and ties on the fifth. Component comparisons, experience-coverage curves, and traced resource decisions characterize how the framework accumulates and uses these resources.

1 Introduction

Spatial reasoning tasks require choosing whether to answer directly or use geometric evidence, while reusable tools alone do not specify when they are useful. SpatialBrain separates current-scene evidence, reusable procedures, and experience about resource use, and improves over direct answers on four of five benchmarks.

1 Introduction

  • Spatial reasoning problems differ: attributes may be read directly, while directions, distances, and routes can require reference frames, multiple views, or movement tracking.Reconstructing every scene can add cost and noisy evidence, while direct answers can miss information that must be integrated across views.Source: §1 · p. 1
  • SpatialBrain retains abstract records of conditions, resources, and execution feedback while keeping model weights fixed, so later tasks can adapt resource choices.Tasks within a block share prior experience, maintain separate scene memories, and update without reference answers.Source: §1 · p. 2
  • 3DFusionMemory fuses object, pose, camera, and route observations into queryable spatial evidence with supporting views and uncertainty.Source: §1 · p. 2
  • The Spatial Skill Library makes recurring spatial computations reusable as parameterized procedures, separating each operation from scene-specific inputs.Source: §1 · p. 2
  • Spatial Cognition associates task conditions with experience about when and how to use memory and skills, using past feedback and costs to inform selection.Source: §1 · p. 2
  • SpatialBrain improves over direct GPT-5.6-sol answers on four of five benchmarks and ties on the fifth.Component comparisons and execution traces examine resource use and how earlier experience changes later decisions.Source: §1 · p. 2

2 Related Work

Related work develops spatial representations, tools, reusable agent workflows, and self-evolving systems. SpatialBrain focuses on using completed test-time experience to adapt spatial resource choices while separating current-scene evidence from reusable procedures and strategies.

Spatial intelligence

  • SpatialVLM and SpatialRGPT use targeted supervision for spatial representations [4] [6], while Thinking in Space studies cognitive maps for video reasoning [27].Source: §2 · p. 2
  • Spatial tools span visual programs, geometric constraints, stateful computation, and simulation [5] [7] [11] [17] [35], while other systems use structured memories, specialist tools, or perspective-aware reasoning [8] [13] [16] [20] [34].SpatialBrain uses execution experience to decide when and how to invoke spatial representations and computations.Source: §2 · p. 3

Agent skills and memory

  • Prior work reuses executable skills and workflows [23] [25] or distills scene-aware tool workflows and failure lessons [14]; SpatialBrain separates procedure, resource-use strategy, and current-scene evidence.It focuses on blockwise, reference-label-free test-time adaptation with fixed model weights.Source: §2 · p. 3

Self-evolving agents

  • Self-evolving agents update different resources, including verbal feedback [21], task experience [32], contextual playbooks [31], agent programs [10], or agent code [30].SpatialBrain updates external spatial strategies and procedural records from completed test-time trajectories.Source: §2 · p. 3

3 SpatialBrain

SpatialBrain separates current-scene evidence from persistent experience about how to reason with it. It fuses observations into queryable relations, reuses parameterized skills, and consolidates completed trajectories between task blocks to guide later resource decisions.

3.1 Overview

  • The reasoner uses a fixed backbone and separates an updating within-task scene graph from a block-shared snapshot of skills and strategic experience.Tasks in a block read the same committed snapshot while keeping separate scene graphs, so reusable procedures persist without carrying earlier object coordinates into a new scene.Source: §3.1 · p. 3

3.2 3DFusionMemory

  • 3DFusionMemory combines multi-view observations in a shared scene frame, linking object identities and computing relations between fused object representations.Queries return distances and camera- or object-frame relations with confidence, supporting views, and ambiguity; ordered route state supports successive movements.Source: §3.2 · pp. 3–4

3.3 Spatial Skill Library

  • Spatial skills are reusable procedures with applicability conditions and input–output interfaces, so the same computation can operate on different task objects.Their parameters bind to current-task objects; the executable configuration can compile compatible primitive calls into procedures under a limited budget.Source: §3.3 · p. 4

3.4 Spatial Cognition

  • Spatial Cognition selects reasoning routes, memory access, and permitted skills from task conditions and prior resource-use experience, subject to support, risk, and cost constraints.The task signature uses visible task properties rather than reference answers or benchmark labels; process feedback tracks reliability and cost, not answer correctness.Source: §3.4 · p. 5
  • Completed trajectories become label-free process records, and the stores update only after a block finishes, so tasks in that block cannot use those updates.The next block can retrieve consolidated strategic and procedural records; tasks within the block can execute in parallel.Source: §3.4 · pp. 5–6
  • Table 1 reports percentage scores across five benchmarks, with an unweighted benchmark mean and a 1,000-question SpaceDG evaluation subset.The table caption identifies evaluation scope but does not provide score values in the supplied passage.Source: Table 1 · p. 6

4 Experiments

SpatialBrain is evaluated against direct answers from the same backbone across five benchmarks, with shared items and specified scoring settings. It improves on four benchmarks and ties on SAT; error patterns vary by task and benchmark, and a trajectory illustrates memory-grounded reference-frame reasoning.

4.1 Experimental Setup

  • SpatialBrain and the direct-answer baseline use GPT-5.6-sol on the same benchmark items; predictions are fixed before scoring, and experience updates use neither reference answers nor future tasks.The evaluation covers MMSI, MindCube, OmniSpatial, SpaceDG, and SAT; Table 1 also includes published comparisons with varying backbones and protocols.Source: §4.1 · p. 6

4.2 Main Results

  • SpatialBrain improves on direct GPT-5.6-sol answers on four benchmarks and ties on SAT, raising the unweighted mean from 67.5 to 72.6.MindCube has the largest gain, 20.4 percentage points, while changes on OmniSpatial and SpaceDG are smaller.Source: §4.2 · p. 6

4.3 Error Analysis

  • MindCube errors fall in all three categories, from 344 to 130 overall, a 62.2% reduction; MMSI has 33 fewer errors, but changes vary by task type.Input-derived groups show the largest reduction for motion and rotation; pooled sequential-reasoning errors decrease on MindCube but increase on MMSI.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4 Memory-grounded Spatial Reasoning

  • In a memory-grounded example, anchors from two views contribute to shared scene memory, and a stair-relative coordinate check resolves the food-storage location.The planner receives the memory atlas, while a tool verifies the stair-relative direction.Source: §4.4 · p. 7

5 Analysis and Discussion

Component comparisons assess the framework on MindCube, while analyses describe what 3DFusionMemory represents and how stored experience and referenced skills accumulate. The reported coverage varies across benchmarks and can grow in task signatures without a comparable increase in distinct skills.

5.1 Component Analysis

  • On 1,050 MindCube items, SpatialBrain scores 87.6, versus 76.7 without the Skill Library, 75.2 without 3DFusionMemory, 78.8 without SpatialCognition, and 67.2 for direct answers.Component-removal gaps are larger on Among and Around than on Rotation; missing and refused answers count as wrong.Source: §5.1 · p. 7, Figure 6 · p. 8

5.2 What Does 3DFusionMemory Represent?

  • Across five evaluation streams, saved 3D objects occur in 50.6% of MMSI tasks and 37.5% of SpaceDG tasks, versus 21.0% for MindCube, 29.2% for OmniSpatial, and 18.0% for SAT.Coverage reflects which inputs produce spatial state, not retrieval or geometric accuracy; complementary observations connect objects in a common scene frame.Source: §5.2 · pp. 7–8

5.3 Experience and Skill Coverage

  • MMSI reaches 165 task signatures and 24 referenced seed skills, while MindCube reaches 12 and 10; on MMSI, task coverage continues growing after skill coverage begins to plateau.Task coverage counts signatures with experience records; skill coverage counts existing seed skill IDs referenced by those records.Source: §5.3 · p. 8

Appendix B.2 separately measures how often earlier experience reaches subsequent tasks in the executable

The appendix describes experience coverage and subsequent exposure, while the conclusion characterizes SpatialBrain as separating current-scene evidence, reusable computation, and resource-use experience. The authors caution that accumulation and observed use do not establish a general improvement from self-evolution.

Appendix B.2 separately measures how often earlier experience reaches subsequent tasks in the executable

  • The experience curves track distinct task signatures with stored records and distinct seed skills referenced by those records across each benchmark’s evaluation scope.The caption describes what is measured but does not report curve values.Source: Figure 7 · p. 9
  • Experience curves and selected trajectories establish accumulation and use, not a general improvement from self-evolution.The authors call for matched evaluations measuring answer quality and the full cost of acquiring and using experience.Source: Appendix B.2 · p. 9

6 Conclusion

  • SpatialBrain separates current-scene evidence, reusable spatial computation, and experience about resource use around a fixed vision-language backbone.Completed trajectories update external experience between task blocks, changing resources and decisions available to later tasks.Source: §6 · p. 9

A Evaluation Settings

The appendix documents evaluation settings and supplementary analyses of stored scene evidence, experience exposure, reasoning categories, and component comparisons. These analyses clarify what the measurements cover and where they do not establish isolated effects.

A Evaluation Settings

  • Published baselines differ in backbones, decoding settings, and tool access, and the SpaceDG subset is sampled with seed 0 from its 9,918-item release.Qwen3-VL-8B’s score summary lacked raw predictions for independent verification; the World2Mind OmniSpatial result includes three failed tool observations.Source: A Evaluation Settings · p. 14, Table A1 · p. 14

B Supplementary Analyses

  • The appendix reports results across distinct evaluation batches, so its coverage, component, and main-table analyses should not be treated as one shared run.The five-stream coverage analyses use an earlier test-time batch; the experience-exposure counts use a later executable configuration.Source: B Supplementary Analyses · p. 15

B.1 3DFusionMemory Coverage

  • MMSI and SpaceDG have higher stored object and relation coverage than other streams, but coverage does not follow a universal 3D-versus-2D split.Object and relation coverage overlap; missing snapshots are unknown rather than observed empty memories.Source: §B.1 · p. 15, Figure A1 · p. 15

B.2 Experience Coverage and Subsequent Exposure

  • Earlier experience reaches 231 of 1,000 MMSI tasks and 212 of 1,050 MindCube tasks, establishing delivery to the reasoner rather than successful execution or improved answers.MMSI forms 18 experience records, six available; MindCube forms five, with three available.Source: §B.2 · p. 15

B.3 Input-Derived Reasoning Categories

  • Pooled error rates are lower for all five input-derived reasoning groups, while benchmark contributions differ and sequential reasoning regresses on MMSI.The category assignment is fixed and retains every item; Measurement has no MindCube items under this taxonomy.Source: §B.3 · p. 16, Table A2 · p. 16

B.4 MMSI Component Comparison

  • The earlier MMSI component study reports 487 Full errors versus 505, 519, and 522 after Skill, Memory, and Cognition removal, respectively.These configurations were not verified as complete removals, and unanswered items account for part of the gaps.Source: §B.4 · p. 17, Figure A2 · p. 17

B.5 MMSI Errors by Task Type

The MMSI task-type analysis reports fewer overall errors for SpatialBrain than Direct, with gains and regressions varying by task type. A separate earlier retrieval analysis measures record retrieval and sequence alignment, not answer improvement.

B.5 MMSI Errors by Task Type

  • 33 fewer MMSI errors overall: SpatialBrain makes 429 errors versus Direct’s 462 on all 1,000 items.Figure A3 compares error rates by task type, with each rate dividing wrong answers by the number of items in that type.Source: §B.5 · p. 18, Figure A3 · p. 18
  • Camera–object errors decrease from 53 to 24 and appearance errors from 39 to 25, while multi-step errors increase from 83 to 94 and region–region errors are unchanged.There are 199 corrections and 166 reversals across the full set.Source: §B.5 · p. 18

B.6 Earlier Retrieval and Sequence Alignment

  • In an earlier GPT-5.5 experiment, 122 of 332 eligible target tasks retrieve a record and 59 execute a partially matching tool sequence; aggregate accuracy and cost do not improve.Partial alignment occurs in 48.4% of tasks with retrieved records; the analysis does not count newly generated skills.Source: §B.6 · p. 18

C Qualitative Analysis and Visualizations

The qualitative section groups trajectories by the roles of memory, reusable skills, and experience-guided resource choices. Selected examples illustrate execution paths, while the paper cautions that these examples and traces do not establish isolated component effects.

C Qualitative Analysis and Visualizations

  • Colored boxes group examples as 3DFusionMemory grounding, Skill Library procedures, SpatialCognition resource choices, or additional tool-based trajectories.The boxes show observed operations rather than isolated component effects, and executing an existing skill differs from learning one.Source: C Qualitative Analysis and… · p. 19
  • Seven of ten detailed trajectories show an incorrect Direct answer and a correct SpatialBrain answer, but the selected cases are not representative of the full set.The full set has 199 corrections and 166 reverse outcomes; overall scores are 53.8% for Direct, 57.4% for the fixed-policy control, and 57.1% for SpatialBrain.Source: C Qualitative Analysis and… · p. 19
  • Among 485 both-correct fixed-policy/SpatialBrain pairs with complete token records, 221 use fewer SpatialBrain tokens and 71 also use fewer calls and less observed time.Elapsed times are observational because concurrency and caches were not controlled.Source: C Qualitative Analysis and… · p. 19

C.1 3DFusionMemory

The examples show 3DFusionMemory connecting detections and camera poses to stored anchors and relations, and the Skill Library returning structured computations for answers. They also specify where stored-view support and the illustrations should not be overinterpreted.

C.1 3DFusionMemory

  • The answer-correction example reconstructs two views into a shared frame, exposes two anchors and one relation, then uses a coordinate check to answer “on the right.”The atlas includes relation uncertainty; the check combines the island observations and uses the stove to define east.Source: §C.1 · pp. 20–21
  • Across the two multi-view examples, 3/5 anchors in one example and 2/6 in the other have support from at least two views; examples were selected for visible geometry, not answer correctness.Colored matrix cells indicate recorded view support, not independently verified object matches; distance scales are not calibrated in meters.Source: Multi-view Support in 3DFusionMemory · pp. 22–23
  • A separate single-view-support analysis finds no anchors supported by two or more views in four additional saved scenes, including one with three retained camera poses.These states illustrate stored layouts rather than repeated-observation fusion.Source: Multi-view Support in 3DFusionMemory · p. 23

C.2 Spatial Skill Library

  • Existing skills return structured outputs for answer generation: a rotation template returns “down,” while a camera-frame direction template returns coordinates resolving a “front left” option.The rotation skill is used at its initial state v0.Source: §C.2 · p. 24

C.3 SpatialCognition

SpatialCognition uses experience from earlier task blocks to adapt whether a spatial skill or direct reasoning is recommended. The traces illustrate both changing skill selection and avoiding unnecessary execution while preserving shared safeguards.

C.3 SpatialCognition

  • The supporting experience excludes the target task, so the recommendation traces draw on earlier task blocks.Source: §C.3 · p. 25
  • With 27 same-family events, the policy changed from recommending agent execution to direct reasoning; the current task followed that recommendation with memory off and no skill execution.The shared 2D-measurement guard remained unchanged.Source: §C.3 · p. 25
  • With 18 earlier same-family events, the policy recommended executing the camera-rotation skill without exploration, while the fixed-policy control also called the skill.The observed change was in selection policy, not skill code or item correctness.Source: §C.3 · p. 25

C.4 Additional Execution Traces

Additional execution traces show that SpatialBrain can use fewer calls and less observed time on some correct-answer paths, while other cases expose cost trade-offs and limits of geometric shortcuts. The appendix also specifies confidence-weighted scene fusion, conservative strategy admission, and which state experience can change.

C.4 Additional Execution Traces

  • For the gripper trace, selected box-center shifts of +57.3 and +17.7 pixels exceeded stationary thresholds of 9.8 and 7.7 pixels, yielding movement of both grippers to the right.Source: §C.4 · p. 26
  • On brick-building and roadside-sign detections in both frames, both methods are correct; SpatialBrain avoids reconstruction and Python calls, while fixed-policy control uses one reconstruction and two Python calls: 88,760→29,415 tokens (−66.9%), 5→2 non-final top-level calls, and 318.6→29.7 s observed.Source: §C.4 · p. 27
  • 90,898 → 52,417 tokens (−42.3%), 9 → 3 non-final top-level calls, and 110.5 → 69.4 s observed on the correct MMSI 880 path.Both runs encounter ambiguous detections; SpatialBrain recovers after an unsuccessful skill attempt, while the fixed-policy run makes eight detector calls.Source: §C.4 · p. 27
  • Correct cross-view height answers do not validate normalized box-center comparisons: the procedure did not estimate alignment or establish a common height reference.The compared normalized heights were 0.792 and 0.825.Source: Reasoning and Cost Boundaries · p. 28
  • 123,214 → 111,609 tokens (−9.4%) came with 4 → 5 calls and 258.9 → 262.3 s observed, so fewer tokens did not mean fewer calls or less time.Source: Reasoning and Cost Boundaries · p. 28

D.1 Confidence-weighted Scene Fusion

  • D.1 fuses each matched object's accumulated center and confidence weight with a view-specific projected center and weight; weights express relative support, not calibrated probabilities.The first valid projection initializes the estimate, and the update requires a positive total weight.Source: §D.1 · p. 29

D.2 Conservative Strategy Admission

  • D.2 admits an alternative only with sufficient earlier, task-signature-conditioned support and complete cost records, comparing its upper cost bound with the default's lower bound.Process-violation tolerance and a required fractional saving are also part of the admission criteria.Source: §D.2 · p. 29

D.3 Learned and Fixed State

  • D.3 fixes the initial library, signature schema, interfaces, checks, and update rules; experience changes support statistics, condition-strategy associations, and procedural availability, not model weights.Process observations do not establish answer correctness; evaluation labels are separate from the update loop.Source: §D.3 · p. 29

Related papers

Loading 2610.29317v1…