PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Memory by Agents, for Agents

Qianwei Wang, Ce Zhang, Zhanpeng Luo, Pengpeng Su, Shuo Yang, Hao Xu

arXiv:2610.22736v1cs.RO

TL;DR

Spatial memory must retain and retrieve diverse evidence for questions that may be unknown during exploration, but designing it for complementary tasks is difficult. The paper uses coding agents to iteratively design memory construction and retrieval, then evaluates frozen systems across five tracks. Codex / GPT-6 Astra scores higher on the equally weighted average than the strongest evaluated full-track baseline, while the paper notes limitations in mechanism analysis and explored system designs.

  • Problem

    Spatial memory must support diverse semantic, metric, and temporal questions, yet retaining evidence for one query type may omit what another requires.

    Source: §I · p. 1

  • Method

    Coding agents iteratively design memory construction and retrieval using public feedback, and the frozen systems are evaluated across five tracks with a fixed Query Agent.

    Source: §I · pp. 1–2

  • Results

    44.23 Unified for Codex / GPT-6 Astra versus 29.32 for ReMEmbR, the strongest evaluated full-track baseline, across three independent searches.

    Source: §I · p. 1

  • Takeaways & Limitations

    Coding-agent design improves average cross-task utility over established baselines, producing compact memories with complementary localization, temporal reasoning, and navigation strengths.

    Source: §VI · p. 7

    Complete-system evaluation leaves individual mechanisms less understood; stronger cloud-hosted reasoners and hybrid designs remain largely unexplored.

    Source: §VI · p. 7

Abstract

from arXiv · show

Spatial memory enables robots to reuse past observations for localization, navigation, and question answering. Designing such memory involves jointly deciding what to retain and how to retrieve it for diverse semantic, metric, and temporal needs. We investigate whether coding agents can automate this design process through iterative implementation and evaluation. Under a fixed development budget, a Designer Agent writes and revises code for memory construction and retrieval using feedback from a fixed Query Agent. The resulting implementations are frozen before building memories from held-out observation histories, without access to downstream questions, and evaluated across five complementary task tracks using the same Query Agent. Across three independent design searches, Codex / GPT-6 Astra averages 44.23 on the equally weighted five-track score, versus 29.32 for ReMEmbR, the strongest evaluated baseline supporting all five tracks, a 50.9% relative improvement. Qualitative analysis relates performance differences to evidence retention and retrieval, while identifying grounding and evidence-use failures. These results demonstrate the potential of coding agents to automate spatial memory design, while revealing task-dependent trade-offs in performance and computational cost. Project page: https://ba-fa-3344.github.io/.

I. INTRODUCTION

Spatial memory must retain and organize complementary semantic, metric, and temporal evidence for questions that may be unknown during exploration. The paper studies whether coding agents can design such memory, reporting a higher five-track average than the strongest evaluated full-track baseline.

I. INTRODUCTION

  • Memory design must serve complementary spatial, semantic, and temporal questions, because retaining information for one query type may omit evidence needed for another.Existing systems use different memory and retrieval designs, and repeatedly implementing and evaluating them makes manual development costly.Source: §I · p. 1
  • A Designer Agent iteratively revises memory construction and retrieval with public-task feedback; the selected implementation is frozen before held-out memories are built without downstream questions.A fixed Query Agent answers through each system’s retrieval tools.Source: §I · pp. 1–2
  • 44.23 Unified versus 29.32 for ReMEmbR: Codex / GPT-6 Astra achieves the higher average across three independent searches on held-out histories.ReMEmbR is identified as the strongest evaluated full-track baseline.Source: §I · p. 1
  • The evaluation covers five tracks: category-level and referring-expression localization, spatial and temporal question answering, and navigation-history question answering.The suite uses separate development and held-out histories with a fixed Query Agent for cross-task comparisons.Source: §I · pp. 1–2
  • The paper contributes a budgeted workflow for jointly designing memory construction and retrieval, and analyzes evidence retention, retrieval, and resource trade-offs.Figure 1 depicts a shared component pool and alternative implementations revised using public feedback.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Related work spans geometric, object-centric, and language-queryable spatial representations, as well as systems for retrieving spatial and temporal evidence. The paper positions its evaluation as a shared build–package–query protocol for coding-agent design.

A. Spatial Memory Construction and Retrieval

  • Spatial-memory work represents geometry, objects, places, and time in varied forms, while retrieval systems expose posed captions, visual snapshots, temporal histories, or relational evidence.Examples include dynamic scene graphs, HOV-SG, VLMaps, CLIP-Fields, OpenScene, ConceptFusion, and ConceptGraphs, –.Source: A. Spatial Memory Construction and… · p. 2
  • Unlike task-list-driven Clio [20] or code generation for map consumption, this work searches memory construction and access during development, then freezes the implementation before held-out builds.The paper focuses on selecting and coordinating established mechanisms.Source: A. Spatial Memory Construction and… · p. 2

B. Functional Evaluation of Spatial Memory

  • NaVQA and OC-NaVQA, are the closest protocol precedents; this suite combines them with constructed questions, OpenEQA, and ScanRefer/ScanEnts3D,.It retains native OC-NaVQA reporting and uses a shared build–package–query evaluation.Source: B. Functional Evaluation of Spatial… · p. 2
  • The suite tests complementary memory capabilities through development feedback and held-out assessment for coding-agent design.Source: B. Functional Evaluation of Spatial… · p. 3

C. Coding-Agent-Driven System Design

The paper frames spatial memory as a jointly designed builder, retrieval interface, and tool-use instructions, then lets coding agents revise these components using public feedback. A fixed development budget and resource-adjusted public score guide the design protocol.

C. Coding-Agent-Driven System Design

  • Coding-agent precedents include architecture search, ML-pipeline refinement, program modification, and robotics program repair; here agents design persistent memory construction and access across tasks.Examples include ADAS, MLE-STAR, the Darwin Gödel Machine, ASPIRE [7], and ENPIRE [8].Source: C. Coding-Agent-Driven System Design · p. 3

A. Spatial Memory as the Design Target

  • A spatial-memory system maps a posed RGB-D history to a memory, retrieval tools, and tool-use instructions, through which a shared Query Agent answers questions.The design target jointly includes the builder, retrieval interface, and instructions.Source: A. Spatial Memory as the Design Target · p. 3
  • The shared module pool includes detection, segmentation, visual-feature, captioning, and multimodal-embedding components; compact modules and a 4B consumer are favored for onboard efficiency.Quantized Qwen3.5-4B supports responsive inference on Jetson Orin devices.Source: A. Spatial Memory as the Design Target · p. 3

B. Designer Protocol

  • Each scratch search begins with an unimplemented builder; the Designer edits memory, retrieval tools, and instructions in an implement–evaluate–inspect–keep/revert–revise loop using only public feedback.The 24-hour budget includes coding, API waiting, building, and evaluation; configurations retain native harness editing and execution.Source: B. Designer Protocol · p. 3
  • The public progress score averages track utility and subtracts package-size and build-time penalties; the time penalty is capped, and query latency is not penalized.The benchmark-composition figure shows query shares, while Unified weights the five tracks equally.Source: B. Designer Protocol · p. 3, Figure 3 · p. 3

C. Frozen Build–Package–Query Evaluation

The implementation is frozen before held-out evaluation, and memories are built from histories without questions or references.

C. Frozen Build–Package–Query Evaluation

  • After freezing, each system builds and reuses packages across tracks without access to questions or references; systems share the Query Agent and evaluation setup, differing only in method-specific tools and instructions.Source: C. Frozen Build–Package–Query Evaluation · p. 3

IV. MULTIFACETED EVALUATION

The evaluation spans five complementary spatial-memory tasks and uses separate development and frozen evaluation histories. Its metrics separately assess localization, semantic QA, temporal precision, and recorded-history navigation answers.

A. Coverage, Sources, and Split

  • Five tracks test category coverage and geometry, instance discrimination, semantic context, temporal evidence, and navigation-history QA.Inputs combine constructed questions with ScanNet, HM3D, ScanRefer, ScanEnts3D, OpenEQA, and OC-NaVQA resources.Source: A. Coverage, Sources, and Split · p. 3
  • 10,347 queries over 176 inputs are split into 1,284 public-development queries and 9,063 frozen-evaluation queries, with complete scene-scan, episode, and trajectory IDs disjoint across splits.Separate captures may revisit physical regions; T5 also includes 177 official frozen questions and 60 constructed public questions on separate trajectories.Source: A. Coverage, Sources, and Split · p. 3

B. What Each Score Measures

  • T1 scores category localization with a soft F-score combining credited localization and instance coverage, while T2 scores only the top-ranked position against the referred target.T1 uses maximum-credit one-to-one matching among the first K predictions and a distance-based credit function.Source: B. What Each Score Measures · p. 4
  • T3 maps a reference-guided semantic correctness rating to [0, 1], while T4 rewards temporal precision relative to the observation horizon and scores missing events as zero.T3's judge uses a separate conversation from the Query Agent but shares its checkpoint.Source: B. What Each Score Measures · p. 4
  • T5 uses exact yes/no accuracy, horizontal-position error, or temporal error for recorded-history questions, and does not measure active navigation success.Scores lie in [0, 1], and missing or malformed answers receive zero.Source: B. What Each Score Measures · pp. 3–4

C. Aggregation and Resource Accounting

The evaluation equally averages supported task scores into Unified and reports deployment resources separately. Baselines share the downstream Query Agent where possible, while retaining specified system components and track-support boundaries.

C. Aggregation and Resource Accounting

  • Unified equally averages the five track scores, partial systems receive no Unified score, and resources are reported separately from performance.Reported resources include build time per frame, package size, peak RAM/VRAM, and end-to-end query latency; the shared Query Agent uses fixed decoding and tool-call settings.Source: C. Aggregation and Resource Accounting · p. 4

A. Setup and Reporting

  • Each of four harness–model configurations runs three independent 24-hour searches, and Avg reports mean ± sample standard deviation across the three search means.Each mean averages three query repeats; Best selects the run with highest public Unified, and frozen evaluation follows selection using public feedback.Source: A. Setup and Reporting · p. 4
  • Baselines share the Qwen3.5-4B Query Agent and common build components where possible, while ReMEmbR uses the same captioner and embedder available to Designers.FARM's captioner and query parser are replaced with Qwen3.5-4B and called FARM-4B; DAAAM retains its own mapping stack, while ConceptGraphs and FARM-4B are evaluated on T1–T3.Source: A. Setup and Reporting · p. 4

B. Effectiveness and Cost

Across three searches, Codex / GPT-6 Astra leads the full-track baseline on Unified, while systems show different task profiles and resource costs. The analyses link those differences to retained evidence, event granularity, and how the Query Agent accesses evidence.

C. Designed Systems and Performance Differences

  • 12.00 of the 16.43-point Unified gap between ReMEmbR and Astra is arithmetically attributable to localization, where observer positions may differ from the described object's location.Astra Best retains object geometry, linked views, and alternative crops; target appearance, full-query context, and conditional target–anchor parsing support instance discrimination.Source: C. Designed Systems and Performance… · p. 4

B. Effectiveness and Cost

  • 44.23 Unified for Codex / GPT-6 Astra versus 29.32 for ReMEmbR, a 14.91-point (50.9% relative) gain across three searches; all 12 systems exceed ReMEmbR on Unified.Astra exceeds DAAAM on all five tracks and FARM-4B on its three supported tracks, but ReMEmbR remains stronger on T5; the Astra public-selected run scores 45.75 frozen Unified.Source: B. Effectiveness and Cost · p. 4, Table I · p. 5
  • 4.31–23.94 MiB mean package sizes for agent-designed systems compare with 67.90 MiB for DAAAM and 67.77 MiB for FARM-4B, while mean build time is 0.143–0.347 versus 0.667/0.425 s/frame.ReMEmbR is smaller at 2.07 MiB; Astra averages 23.94 MiB and 25.93 s/query versus Opus 5 at 4.31 MiB and 14.74 s/query.Source: B. Effectiveness and Cost · p. 4, Table II · p. 5

C. Designed Systems and Performance Differences

  • 56.39 T4 for Claude Opus 5 Best versus 31.38 for DAAAM is consistent with task-facing distinctions among sightings, encounters, passing, area visits, and level arrivals.Astra Best retains temporal evidence but is stronger on T1–T3 and weaker on T4/T5, reflecting complementary evidence demands.Source: C. Designed Systems and Performance… · p. 5
  • DeepSeek's seven-tool Run 2 scores 0.75 points lower on Unified while requiring almost three times the build TPF, although it improves T2–T4.More components offer capacity only when their evidence remains usable within the shared interaction budget.Source: C. Designed Systems and Performance… · p. 5
  • Even the best T1/T2 scores are 22.33/39.14, and precise temporal answers can describe the wrong event; selected systems also fail frozen queries through context overflows and invalid tool arguments.The analysis identifies challenges in both memory representation and the agent's use of evidence.Source: C. Designed Systems and Performance… · pp. 5–6

D. Search Process and Evidence Use

Development traces show that changing how retained evidence is presented can improve answers, while adding an access step can leave existing evidence unused. A separate figure compares resource-adjusted progress across searches.

D. Search Process and Evidence Use

  • Figure 4 tracks resource-penalized public Unified best-retained progress across three searches using eight H100 GPUs per search.Between-system variation exceeds fixed-system query variation.Source: D. Search Process and Evidence Use · p. 6
  • For a 19.7 s living-room duration, promoting visit spans changes the answer from 113.3 s to 13 s, and stationary-camera support improves it to 15.333 s, with the same build and tool.The answer remains imperfect; the example illustrates improved evidence use through readout changes.Source: D. Search Process and Evidence Use · p. 6
  • Across 1,284 public queries, the Query Agent makes no extra coordinate-inspection call, T2 falls from 35.49 to zero, and the revision is reverted despite geometry being present.Figure 2 illustrates complementary refinement of stored semantic evidence.Source: D. Search Process and Evidence Use · p. 6

E. Seeded Initialization

Seeded memories had mixed success extending coverage without losing strengths, while OC-NaVQA results varied with the answerer. Codex / GPT-6 Astra Best improved substantially with GPT-5-mini, but ReMEmbR retained the best position and time errors.

E. Seeded Initialization

  • 44.22 Unified: one ReMEmbR-seeded search largely preserves T3, improves T5, and exceeds the matched scratch Best at 42.46 Unified.Both ReMEmbR-seeded outcomes improve T1, T2, and T4; the other outcome loses T3/T5 performance.Source: E. Seeded Initialization · p. 6, Table III · p. 7
  • Both FARM-4B-seeded outcomes add T4/T5 support but remain below the matched scratch Avg of 40.66.Run 1 improves only T2 on FARM-4B’s original T1–T3, while Run 2 improves all three.Source: E. Seeded Initialization · p. 6, Table III · p. 7

F. OC-NaVQA and Answerer Compatibility

  • Codex / GPT-6 Astra Best rises from 59.70% with Qwen3.5-4B to 77.61% with GPT-5-mini, while ReMEmbR leads position/time errors at 24.84 m/1.75 min.Claude / Claude Opus 5 Best leads binary accuracy with both answerers (78.61/79.60%); Astra lowers DAAAM’s errors to 44.32 m/2.36 min.Source: F. OC-NaVQA and Answerer Compatibility · p. 6, Table IV · p. 7

G. Simulation Robot Deployment

In a ProcTHOR house simulation, a robot used prebuilt, read-only memories for navigation and question answering. Agent-designed memory helped navigation, while QA remained comparable to ReMEmbR and target confirmation was a major difficulty.

Simulation setup

  • The 40 house tasks comprise 20 navigation goals and 20 QA questions; systems share a Qwen3.5-4B controller, task starts, and prebuilt memories.Memories are built from one robot survey and read-only during evaluation.Source: G. Simulation Robot Deployment · p. 7

Turning memory into navigation targets

  • 58.3% navigation success: agent-designed memory outperforms ReMEmbR’s 10.0%, with object coordinates and crops providing actionable targets.In 20 of 25 failed episodes, the robot declared success prematurely; five more failed depth-confirmed visibility near the target.Source: G. Simulation Robot Deployment · p. 7

Context and coverage in memory QA

  • 33.3% QA versus ReMEmbR’s 36.7%; ReMEmbR favors distance comparisons, while both systems struggle with complete, unique counts.Distance comparisons favor ReMEmbR (9/18 versus 7/18); bounded retrieval and instance/context ambiguity can impede enumeration.Source: G. Simulation Robot Deployment · p. 7

VI. DISCUSSION

The discussion concludes that spatial memory benefits from coordinated retention, retrieval, and use, with coding-agent design improving average cross-task utility. It also identifies limits in mechanism-level understanding and the systems explored.

Conclusion

  • Coding-agent design improves average cross-task utility over established baselines, with complementary localization, temporal reasoning, and navigation strengths.The conclusion emphasizes compact memories and coordination among retention, retrieval, and use.Source: §VI · p. 7

Limitations

  • Complete-system evaluation leaves individual mechanisms less understood; stronger cloud-hosted reasoners and hybrid designs remain largely unexplored.The authors call for controlled ablations and note answerer-dependent OC-NaVQA results and the need for independent T3 judges.Source: §VI · p. 7

Future Work

  • The authors propose pairing task metrics with public feedback, held-out environments, explicit budgets, and agent-designed baselines to evaluate robotics systems.They frame this as a way to test mechanisms, component coordination, and efficiency under changing demands.Source: §VI · p. 8
Loading 2610.22736v1…