PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen

arXiv:2608.12743v1cs.AI

TL;DR

Spatial reasoning remains challenging for current VLMs, motivating parameter-update-free self-evolution for frozen agents without external expert spatial tools at inference. SMA distills verified experience into transferable lessons and retrieves them using calibrated reliability scores, achieving the best accuracy among evaluated methods in most of 20 evaluations across five benchmarks and four base VLMs.

  • Problem

    Spatial reasoning remains challenging for current VLMs, motivating study of parameter-update-free self-evolution for frozen agents without external expert spatial tools at inference.

  • Method

    SMA uses verifier-guided reflection to distill verified spatial experience into transferable lessons, then retrieves them through semantic similarity and calibrated Transfer Reliability Scores.

  • Results

    SMA achieves the best accuracy among evaluated methods in most of 20 evaluations across five representative spatial benchmarks and four base VLMs.

  • Takeaways & Limitations

    External transferable lesson memory offers a practical parameter-update-free route for spatial self-evolution that complements training-based post-training and tool-augmented reasoning.

  • Takeaways & Limitations

    TRS analysis is limited by differences in benchmark difficulty and question composition.

Abstract

from arXiv · show

Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.

1 Introduction

Spatial reasoning remains challenging for current VLMs, motivating parameter-update-free self-evolution through reusable lessons from verified spatial experience. SMA keeps the VLM frozen, uses verifier-guided memory and transfer-reliability retrieval, and achieves the best accuracy in most evaluated benchmark-model evaluations.

  • 1 Introduction: Although spatial intelligence is increasingly important, recent VLM progress and benchmarks show that spatial reasoning remains challenging, while existing improvements mainly use post-training or external spatial tools.SMA studies the underexplored alternative of parameter-update-free self-evolution focused on transferring lessons rather than memorizing solved instances.
  • 1 Introduction: SMA is an experience-grounded runtime framework that keeps VLM parameters frozen while converting verifier-grounded spatial experience into reusable transferable lessons.For each problem, it queries the frozen VLM, obtains a predicted answer and reward, and applies verifier-guided reflection.
  • 1 Introduction: SMA retrieves lessons using task-embedding semantic filtering and similarity-TRS ranking, with TRS calibrated from later visit evidence of transfer reliability.The retrieved procedures guide read-only deployment on new spatial tasks without external expert spatial tools at inference time.
  • 1 Introduction: Across five spatial benchmarks and four base VLMs, SMA achieves the best accuracy among evaluated methods in most of the 20 evaluations.This establishes external transferable lesson memory as a practical parameter-update-free path for spatial self-evolution.

2 Related Work

Related work mainly pursues spatial post-training and self-evolving or memory-augmented agents. SMA addresses gaps in these routes through training-free spatial self-evolution and a Transfer Reliability Score for spatial procedure transfer.

  • Existing spatial-reasoning work mainly explores two routes, including spatial post-training; SpatialVLM uses instruction data to teach object relationships and spatial grounding (Chen et al., 2024).
  • Self-evolving and memory-augmented agents reuse interaction traces, reflective reward, retrieved knowledge, and long-term state to improve later decisions without training on each new task (Zhong et al., 2024; Park et al., 2023; Shinn et al., 2023).
  • A key gap is that existing approaches do not distinguish surface relevance from whether a spatial reasoning procedure has reliably transferred to new contexts.
  • SMA addresses both gaps with training-free self-evolution and a Transfer Reliability Score that estimates spatial procedure-transfer reliability across new visual contexts and tasks.During memory writing, reflection compresses verified rollouts into procedural memories; read-only deployment retrieves candidates using semantic similarity and TRS while keeping parameters and the memory bank unchanged.

3 Method

SMA enables parameter-update-free spatial self-evolution by converting verifier-grounded experience into transferable procedural memories and using them to guide a frozen VLM. It separates experience acquisition with memory calibration from read-only deployment using semantic and reliability-aware retrieval.

  • Overview: SMA builds an external library of transferable spatial procedures to improve deployment problems without changing the base model’s parameters.The environment and deployment splits are disjoint, and the memory bank guides new reasoning only during read-only deployment.
  • Memory Construction: Each memory card stores a task, rollout summary, transferable lesson, visit statistics, accumulated reward, and Transfer Reliability Score.Retrieved cards expose only the task, summary, and lesson, not prior predictions or verified answers.
  • Memory Construction: Verifier-guided reflection converts scored rollouts into strict-JSON summaries and lessons describing task patterns, failure traps, and checks without restating answers.The main method provides the verified target to reflection while enforcing anti-leakage rules.
  • Retrieval: Two-stage retrieval filters cards by semantic similarity, then ranks candidates using normalized similarity and TRS before prepending the top-k guidance cards to the prompt.The combined score reduces superficial over-ranking by incorporating calibrated transfer reliability, with η controlling the TRS weight.
  • Reliability Calibration: TRS starts uniformly and is calibrated from subsequent retrieval visits and rewards, shrinking low-visit memories toward the shared prior rather than reflecting their creation rollout’s correctness.One-Pass Memory Writing creates cards only on the first environment pass; later passes reuse the fixed bank to update reliability.

4 Experiments

Across five spatial benchmarks and four frozen VLMs, SMA achieves the best macro average in every base-model block while improving performance through structured memory, calibrated retrieval, and transferable experience. Additional analyses show that TRS-guided memories transfer across models and benchmarks, improve broad spatial abilities, and enable more efficient memory writing.

  • Experimental Setup: The experiments span RoboSpatial, ERQA, Omni3D, SAT, and EmbSpatial, using Qwen3.5-9B, Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, and Qwen3.6-27B as frozen base models.The benchmarks cover embodied robot perception, physical scene understanding, 3D relations, abstract spatial aptitude, and instruction-grounded embodied reasoning.
  • Overall Results: SMA achieves the best macro average in every base-model block: 68.8, 66.7, 69.8, and 63.5, with gains of 2.6, 2.9, 1.7, and 2.8 points over the strongest non-SMA baselines, respectively.The evaluation covers five benchmark slices and four frozen base models, with the full comparison reported in Table 1.
  • Component Ablations: Removing structured summaries, transferable lessons, semantic filtering, raw outputs, or reward-only reflection lowers RoboSpatial accuracy by 3.2, 3.5, 5.8, 4.4, and 5.5 points, respectively.These ablations use Qwen3.6-27B and isolate memory representation, reflection signal, and retrieval components.
  • Transfer and Reliability: Transferred memories improve accuracy across models and benchmarks, while memories from successful source questions yield 24.3 percentage points higher evaluation accuracy.Accuracy rises from 19.3% for mean TRS in [0.2, 0.3) to 97.3% in [0.9, 1.0], although the pooled trend may reflect benchmark difficulty and question composition.
  • Retrieval and Efficiency: SMA reduces macro-average retrieved-memory similarity from 0.792 to 0.698 while raising macro accuracy from 66.8% to 69.8%, showing that nearest-neighbor retrieval is insufficient.One-Pass Memory Writing uses one-tenth as many memories, reduces redundancy by 21%, and roughly doubles TRS-update coverage by the final pass.
  • Ability-Level Analysis: SMA improves all ten evaluated atomic abilities on average, with largest gains of +11.2 pp for Correspondence, +8.0 pp for Attribute, and +7.6 pp for Object motion.Qualitative cases show retrieval of procedures for size checking, coordinate localization, depth comparison, motion simulation, and background anchoring.

5 Conclusion · Appendix

SMA is an experience-grounded, training-free framework that converts verified spatial experience into reusable transferable lessons for frozen VLM agents. Across five spatial benchmarks and four base VLMs, it achieved the best accuracy among evaluated methods in most of 20 evaluations.

  • 5 Conclusion: SMA provides an experience-grounded, training-free framework for frozen VLM agents.
  • 5 Conclusion: The framework converts verified spatial experience into reusable transferable lessons.
  • 5 Conclusion: Across five representative spatial benchmarks and four base VLMs, SMA achieved the best accuracy among evaluated methods in most of the 20 evaluations.
  • 5 Conclusion: SMA avoids updating model weights while supporting frozen-agent improvement through runtime memory.
  • 5 Conclusion: Verifier-guided reflection distills reusable lessons from verified spatial experience instead of merely replaying past instances.
  • 5 Conclusion: Transfer Reliability Scores are calibrated from visit evidence, and reliable lessons are retrieved to guide later inference.

A More Experiment Results · A.1 Additional Results on SITE-image and ViewSpatial

Table 7 extends the evaluation to SITE-image and ViewSpatial across all four base models and compared baselines, reporting accuracy and macro-average results. It provides per-column best and second-best comparisons for the two benchmarks.

  • A More Experiment Results: Table 7 reports extended results for SITE-image and ViewSpatial across all four base models.
  • A.1 Additional Results on SITE-image and ViewSpatial: The evaluation includes all compared baselines for every base model.
  • A.1 Additional Results on SITE-image and ViewSpatial: Results are reported using accuracy measured in percentage points.
  • A.1 Additional Results on SITE-image and ViewSpatial: The table includes an Avg. column defined as the macro average over the two benchmark columns.
  • A.1 Additional Results on SITE-image and ViewSpatial: The benchmark columns separately cover SITE-image and ViewSpatial.
  • A.1 Additional Results on SITE-image and ViewSpatial: Table 7 identifies the best and second-best result per column.

A.2 More Ablation and Hyperparameter Sensitivity Results

This section extends the ablation analysis to Omni3D using Qwen3.6-27B, matching the main-paper RoboSpatial ablation format and examining hyperparameter sensitivity.

  • Ablations and reflection settings: The Omni3D ablation table uses Qwen3.6-27B additional tuning runs and compares omitted components and reflection variants against the SMA reference.The reporting format and ablation settings match the main-paper RoboSpatial ablation.
  • Hyperparameter sensitivity: Figure 7 presents the Omni3D hyperparameter sensitivity analysis.

B Method Details · B.1 Design of Visit-Evidence Calibration · B.2 SMA Pseudocode

SMA calibrates each memory’s Transfer Reliability Score from downstream visit evidence using a conservative prior, then combines reliability with semantic similarity for retrieval. Its pseudocode separates experience-building, memory updates, and read-only deployment without adding memories or updating scores during deployment.

  • B.1 Design of Visit-Evidence Calibration: Calibration is order-invariant and evidence-driven: memories with equal visit counts and cumulative rewards receive equal scores, while reliable lessons are promoted and harmful ones suppressed as visits accumulate.The design avoids overreacting to one or two noisy visits and lets downstream transfer outcomes determine usefulness beyond the originating problem.
  • B.1 Design of Visit-Evidence Calibration: SMA initializes every memory with neutral TRS and updates it from cumulative verifier rewards over later retrievals, rather than trusting the originating rollout.Each memory stores visit count n_i, cumulative reward c_i, and TRS v_i, initialized as n_i = 0, c_i = 0, and v_i = v_0.
  • B.1 Design of Visit-Evidence Calibration: At retrieval, SMA first filters memories by semantic similarity, then ranks candidates by normalized similarity combined with TRS to balance topical relevance against empirical reliability.Z-score normalization only puts the two ranking terms on a comparable candidate-set scale; it does not replace λ-based low-visit shrinkage.
  • B.2 SMA Pseudocode: During experience collection, SMA shuffles environment problems across passes, retrieves eligible memories, generates frozen-VLM predictions, scores them, updates selected memories, and reflects on the first pass to write new memories.The procedure takes a frozen VLM, reflection model, environment problems, deployment problems, retrieval size, similarity threshold, reliability weight, initial value, and prior strength as inputs.
  • B.1 Design of Visit-Evidence Calibration: The TRS update is a prior-smoothed empirical success rate, v_j = (λv_0 + c_j)/(λ + n_j), so low-visit memories change conservatively while repeated evidence dominates.The prior acts like λ virtual visits with average reward v_0; larger λ makes calibration more stable, whereas smaller λ makes it more reactive.
  • B.2 SMA Pseudocode: In read-only deployment, SMA retrieves and uses existing memories for each deployment problem, saves predictions, and performs no new memory writing or updates to visit counts, rewards, or TRS.The deployment loop returns predictions together with the accumulated memory bank after inference.

C Experiment Details … C.2.4 EmbSpatial

The experiments span seven complementary spatial benchmark slices and organize four main-paper benchmarks into ten atomic capabilities, with detailed category annotations for each benchmark. The appendix further documents benchmark-specific subcategories and representative examples, including EmbSpatial’s directional and distance relations.

  • C.1 Benchmark Overview: Seven benchmark slices cover complementary input formats, answer spaces, and spatial reasoning demands, with RoboSpatial, ERQA, Omni3D, SAT, and EmbSpatial in the main table and SITE-image and ViewSpatial in the appendix.The evaluated slices span robot-oriented understanding, embodied question answering, 3D reasoning, dynamic spatial aptitude, language-grounded relations, image-only spatial tasks, and multi-perspective localization.
  • C.1.1 RoboSpatial; C.1.2 ERQA: RoboSpatial tests normalized free-space localization, object configurations, and placement compatibility, while ERQA emphasizes robot-relevant state, action, trajectory, and spatial facts.RoboSpatial uses single RGB images; ERQA evaluates whether answers support physical robot decisions such as selecting actions or tracking state changes.
  • C.1.3 Omni3D; C.1.4 SAT: Omni3D requires open-answer reasoning about 3D quantities and relations, whereas SAT evaluates binary questions involving goals, actions, perspectives, object movement, and egocentric movement.Omni3D includes metric estimates, occlusion, containment, capacity, and counterfactual placement; SAT uses one or two ordered still images to stress dynamic and egocentric reasoning.
  • C.1.5 EmbSpatial; C.1.6 SITE-image; C.1.7 ViewSpatial: EmbSpatial evaluates language-grounded left, right, above, under, close, and far relations, while SITE-image and ViewSpatial cover image-only spatial tasks and multi-perspective localization.EmbSpatial complements RoboSpatial with a larger image QA pool; SITE-image retains only SITE’s image examples, and ViewSpatial uses multiple scene views.
  • C.2 Benchmark Taxonomy and Atomic Capability Annotation: The taxonomy aggregates heterogeneous benchmark results into ten atomic capabilities: Correspondence, Attribute, Object motion, Localization, Relation, Distance/depth, Mental simulation, Tracking, Camera reasoning, and Affordance.Benchmark categories may map to multiple capabilities when they require compound reasoning procedures.
  • C.2 Benchmark Taxonomy and Atomic Capability Annotation: The main-paper atomic-ability breakdown covers RoboSpatial, ERQA, SAT, and EmbSpatial, while excluding Omni3D, SITE-image, and ViewSpatial because suitable question-level taxonomies are unavailable or outside the reported statistics.Omni3D annotations expose answer_type values float, int, and str rather than spatial-ability categories.
  • C.2.1 RoboSpatial; C.2.2 ERQA: RoboSpatial and ERQA receive detailed category annotations covering context, configuration, compatibility, spatial reasoning, trajectory reasoning, action reasoning, state estimation, pointing, multi-view reasoning, and task reasoning.Representative examples and annotations are reported in Tables 9–18.
  • C.2.3 SAT; C.2.4 EmbSpatial: SAT’s taxonomy includes Goal Aiming, Action Consequence, Allocentric Perspective, Action Sequence, Object Movement, and Egocentric Movement, while EmbSpatial annotates directional and distance subcategories with examples in Tables 25–26.SAT categories capture target-facing direction, action-induced spatial changes, observer-relative relations, frame-to-frame movement, and ego movement.

C.3 Dataset Split Construction … C.6 Hyperparameter Settings

The paper constructs disjoint environment/deployment splits, compares progressively different memory baselines, and evaluates SMA as a parameter-update-free procedural-memory method. Experiments use specified computing infrastructure and backbone-specific hyperparameter tables.

  • C.3 Dataset Split Construction: The retained non-SAT image-question pools are split 50/50 by category with seed 42 into disjoint environment splits for memory writing and deployment splits for read-only evaluation.For odd-sized categories, the extra example alternates between splits, starting with environment; Table 27 reports the original and retained pools.
  • C.4.1 No memory: No memory freezes the VLM and evaluates it directly on the deployment split with the benchmark prompt, without environment-stage writing or external-memory retrieval.This provides the direct frozen-model baseline.
  • C.4.2 RAG: RAG (Lewis et al., 2020) stores prior rollout records and retrieves task-similar examples with prior outputs, but performs no reflected lesson writing and uses no TRS.RAG is therefore an example-retrieval baseline rather than a procedural-memory method.
  • C.4.3 MemP: MemP (Fang et al., 2026) reflects rollouts into compact summaries and transferable lessons, then retrieves them by semantic similarity without visit-evidence reliability calibration.It isolates the contribution of reflected procedure text.
  • C.4.4 MemRL-R / C.4.5 MemRL-GT: MemRL-R and MemRL-GT (Zhang et al., 2026) vary reflection supervision: MemRL-R uses model outputs and scalar verifier rewards, whereas MemRL-GT uses ground-truth-guided feedback.MemRL-GT separates stronger reflection supervision from memory-value calibration.
  • C.5 Computing Infrastructure: Experiments run on a Linux server with four NVIDIA H200 GPUs, dual 48-core Xeon Platinum 8558 CPUs, 192 logical threads, and 2.0 TiB system memory.The reported software stack includes Ubuntu 22.04.5 LTS, CUDA 12.8, Python 3.12.13, PyTorch 2.11.0+cu128, Transformers 5.8.1, and vLLM 0.20.0.
  • C.6 Hyperparameter Settings: Hyperparameter settings are reported in one compact table for each backbone: Qwen3.5-122B-A10B, Qwen3.6-35B-A3B, Qwen3.6-27B, and Qwen3.5-9B.Tables 28–31 provide the selected SMA settings for these four backbones.

D Limitations … E.1 Successful Cases

SMA’s limitations concern unresolved credit assignment and long-term memory maintenance, while qualitative analyses show that retrieved transferable procedures support correct spatial reasoning across diverse tasks and benchmarks.

  • D.1 Credit Assignment in Memory Evolution: SMA cannot precisely attribute outcome changes to memory writing, reflection, retrieval, semantic filtering, or final memory use.Its task-level verification feedback only indicates whether a retrieved or newly written memory should be reinforced, revised, or discarded.
  • D Limitations: SMA’s broader limitation is that it addresses transferable-lesson writing, retrieval, and reliability weighting without implementing a full long-term memory maintenance lifecycle.As the memory bank grows, maintenance becomes important because lessons can become redundant, conflicting, overly specific, or stale.
  • D.2 Long-Term Memory Maintenance: SMA lacks explicit lifecycle management for deleting, merging, compressing, expiring, or rewriting memories under storage or latency constraints.TRS can downweight unreliable memories and semantic filtering can remove weak retrievals, but lessons may still become redundant, conflicting, overly specific, or stale.
  • E.1 Successful Cases: The qualitative analysis examines six successful-transfer and eight wrong-to-right cases spanning all seven benchmarks.The cases test whether reusable reasoning procedures, rather than copied answers, help resolve new visual evidence.
  • E.1 Successful Cases: In successful-transfer cases, both the baseline and SMA answer correctly after retrieving a relevant transferable lesson, whereas wrong-to-right cases show SMA correcting a baseline error after retrieval.The cases cover 3D spatial, camera-motion, temporal and relative-depth, robot-scene, viewpoint, and embodied spatial reasoning.
  • E Qualitative Results: The qualitative results provide cross-benchmark evidence that memory-guided reasoning can transfer across distinct spatial environments and deployment tasks.The analyzed cases span Omni3D, SAT, SITE-image, ERQA, ViewSpatial, RoboSpatial, and EmbSpatial.
  • E.1 Successful Cases: Each qualitative panel traces the question and image, retrieved memory and final TRS, model response, and alignment between the lesson and current visual evidence.This layout assesses whether the retrieved procedure supports the model’s decision rather than merely supplying an answer.

E.2 Failure Cases … ViewSpatial: Memory retrieval prompt

The paper characterizes failure sources, showing that ambiguity in benchmark specifications can make answers underdetermined while persistent visual-grounding errors remain despite relevant memories. Its prompt templates operationalize transferable, shape-matched procedural memories that guide reasoning without copying prior answers, while requiring re-derivation from current visual evidence.

  • E.2 Failure Cases: SMA failures separate benchmark-side ambiguity from base-model limitations: the former leaves answers underdetermined, whereas the latter persists despite relevant retrieved memories.Benchmark ambiguity includes underspecified questions, missing metric or depth information, partial evidence, and hidden simulator geometry; model limitations include misreading movement direction, object count, or spatial connectivity even with TRS ≥0.6 memories.
  • E.2 Failure Cases: Failure examples show that retrieved procedures cannot compensate for ambiguous poses, unspecified depth definitions, unavailable measurements, or inaccurate visual grounding.Examples include depth-ranking errors caused by image-position cues, clearance judgments based on generic object dimensions, and viewpoint answers that depend on an undefined arrival heading.
  • F Prompt Templates: The appendix reports three prompt classes per benchmark: task-solving system prompts, reflection prompts that create transferable lessons, and retrieval prompts that prepend selected memories to later tasks.Runtime memory blocks contain rank, similarity, task, transferable lesson, and summary values, and are omitted when no memory is retrieved.
  • RoboSpatial: System prompt: RoboSpatial prompts organize reasoning around vacant-region pointing, object-configuration verification, and placement compatibility, while enforcing current-image grounding, shape-specific checks, and exact answer formats.The protocol requires multiple points in unoccupied normalized coordinates, literal relation verification, and clearance or obstacle checks; current image evidence overrides memory.
  • RoboSpatial: Reflection prompt: RoboSpatial reflection and retrieval prompts convert rollout diagnoses into abstract lessons for the same question shape, prohibit answer leakage, and require re-deriving each new answer from the current image.Retrieved memories provide checks or traps rather than answer keys, with separate handling for pointing, configuration, and compatibility questions.
  • Reflection and memory retrieval prompts: The prompt design therefore treats memory as procedural guidance for new instances that share a structural shape, not as an answer key for a repeated scene.This principle is explicitly stated for future-agent reasoning and is paired with anti-leakage constraints in the reflection prompts.
  • RoboSpatial: Memory retrieval prompt: Across benchmark prompts, memory retrieval is structural rather than noun-based: applicable memories contribute limited checks or traps, while prior coordinates, labels, letters, and wording are never copied.The supplied RoboSpatial and ERQA retrieval prompts both instruct the agent to ignore mismatched memories and re-derive outputs from current attachments.
  • ERQA: System prompt: ERQA prompts cover kinematics, trajectories, task states, multi-view correspondence, and surface pointing, using attachment-order alignment and shape-specific visual validation before selecting one answer letter.Reflection and retrieval instructions emphasize procedural lessons, anti-leakage, and checks such as simulating motion, verifying containment, and matching cross-view geometry rather than colors.
Loading 2608.12743v1…