PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSight: Task-Driven Self-Improving Memory for Embodied Agents

Lirong Che, Yang Chen, Yimeng Su, Daqi Liu, Zhenyu Huang, Jiong Wang, Yuzhe Yang, Xu Cao, Chuang Wang, Jian Su, Xueqian Wang

arXiv:2610.33487v1cs.ROcs.CV

TL;DR

Robots may have relevant historical observations but lack tools to carry out the operations a query requires. ReSight combines reprocessable spatiotemporal evidence with Explore–Distill–Mount, which turns exploration-derived solutions into reusable capabilities. With the execution model and online budget fixed, EDM improves localization and question answering on the reported benchmarks.

  • Problem

    Existing tools may not support the operations needed to find, interpret, and combine relevant evidence already present in retained observations.

    Source: §1 · p. 2

  • Method

    ReSight retains spatiotemporal observations separately from extensible derived fields, and EDM distills execution and exploration records into reusable tools, skills, and memory builders.

    Source: §1 · p. 2

  • Results

    With the execution model and online budget fixed, EDM reduces OC-NaVQA mean position error from 33.10 m to 24.72 m and raises FindingDory success from 38.3% to 63.8%.

    Source: §4.3 · pp. 6–7

  • Takeaways & Limitations

    Experiments demonstrate effective memory access and capability reuse across held-out episodes and new robot memories, with fewer online tool calls.

    Source: §1 · p. 3, §6 · p. 9

    The predefined query sets and task categories only partially capture the evolving long-tail queries of real-world deployment.

    Source: §5 · p. 9

Abstract

from arXiv · show

Robots can use historical observations to answer spatiotemporal queries in familiar environments, but the presence of relevant evidence does not imply that existing tools support the operations a task requires. We introduce ReSight, a task-driven self-improving memory system for embodied agents that expands the capabilities for processing and accessing historical observations as task demands arise. ReSight retains keyframes and their spatiotemporal associations, separating observation evidence from extensible derived fields so that stored observations can be reprocessed for new needs. An execution agent and a coding agent share this memory through different interfaces: the former calls registered tools within a bounded online budget, while the latter uses a programmable workspace and a larger exploration budget to inspect evidence, write programs, and acquire models as needed. To realize capability expansion in ReSight, we develop Explore-Distill-Mount (EDM), which identifies capability gaps from execution and exploration records without task reference answers and converts exploration-derived solutions into reusable memory builders, query tools, and skills, extending memory representations and access capabilities as needed. Experiments on three public benchmarks demonstrate effective base memory access and capability expansion. With the execution model and online budget held fixed within each benchmark, EDM reduces mean position error on OC-NaVQA from 33.10 m to 24.72 m and improves high-level success on FindingDory from 38.3% to 63.8%. With the tools and skill frozen, success still improves by 28.3 percentage points on FindingDory episodes excluded from exploration and distillation. Experiments on real robot patrol memories further demonstrate reuse of these capabilities for subsequent queries and new memories, with fewer online tool calls on average.

1 INTRODUCTION

ReSight addresses tasks where relevant evidence is stored but existing tools cannot perform the operations needed to answer queries within the online budget. It keeps observations reprocessable and expands capabilities by distilling exploration into reusable tools, skills, and memory builders.

1 INTRODUCTION

  • Historical observations can guide where robots go, but queries about changed objects also require distinguishing observations from different times.The paper focuses on finding, interpreting, and combining evidence already present in retained observations.Source: §1 · p. 2
  • Existing memory approaches can retrieve relevant observations, yet available tools may not support operations such as reading similar door signs or organizing object events across time.These unmet operations create capability gaps when the executor cannot reliably perform them with its existing tools and online budget.Source: §1 · p. 2
  • ReSight retains keyframes and spatiotemporal associations while allowing stored observations to be reprocessed into new fields, tools, and memory builders as task needs arise.Initial semantic fields support retrieval; later processing can derive text, event, or geometric fields linked to source observations.Source: §1 · p. 2
  • The execution agent uses registered tools within a bounded online budget, while the coding agent explores the same evidence through a programmable workspace and a larger budget.The coding agent can inspect files and images, write and execute code, and acquire models or dependencies.Source: §1 · p. 2
  • EDM uses execution and exploration records to identify capability gaps without task reference answers, then packages exploration-derived solutions as reusable tools, skills, and memory builders.Builders produce persistent fields and indices; query tools and skills are mounted for execution-agent use.Source: §1 · p. 2
  • With execution model and online budget fixed within each benchmark, EDM improves OC-NaVQA spatial localization and FindingDory question answering; with tools and skill frozen, success still improves by 28.3 percentage points on excluded episodes.Real-robot experiments demonstrate reuse for subsequent queries and new memories.Source: §1 · p. 3

2 RELATED WORK

Prior work represents spatial semantics, long-horizon visual memories, and programmatic agent capabilities in varied ways. ReSight builds on these directions by retaining reprocessable sensor evidence and extending memory processing and access as needs arise.

2.1 SPATIAL REPRESENTATIONS AND TARGET LOCALIZATION

  • Spatial representations include language-aligned maps, semantic fields, multimodal memories, 3D instances, scene graphs, and spatiotemporal representations for environmental change.Task-driven representations can adjust semantic granularity according to predefined tasks or subsequent queries.Source: §2.1 · p. 3

2.2 LONG-HORIZON VISUAL MEMORY AND RETRIEVAL

  • Long-horizon visual memories retain descriptions, events, tracks, snapshots, embeddings, or keyframes, while retrieval and fusion select relevant information for queries.ReSight retains sensor observations as reprocessable evidence and adds structured fields as subsequent needs arise.Source: §2.2 · p. 3

2.3 PROGRAMMATIC PROBLEM SOLVING AND HARNESS EVOLUTION

  • Programmatic agents and harness-evolution methods reuse code, accumulate skills, or update tools, runtime mechanisms, memory construction, and retrieval programs.EDM synthesizes tools, skills, and memory builders from execution-side capability gaps and coding-agent exploration traces for reuse during online execution.Source: §2.3 · p. 3

3 METHOD

ReSight stores spatiotemporal evidence separately from extensible derived fields and provides asymmetric interfaces for bounded execution and open-ended exploration. Explore–Distill–Mount converts exploration solutions into capabilities that the execution agent can reuse.

3 METHOD

  • ReSight provides a shared spatiotemporal evidence memory with extensible derived fields, accessed through distinct interfaces for execution and coding agents.EDM implements capability expansion within this architecture.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM SETTING

  • Tasks assume their required information is present in retained observations, but expose a capability gap when current tools, skills, or memory access cannot solve them reliably within budget.For example, searching images individually for a visible room sign may exhaust the call budget, motivating dedicated text extraction and retrieval.Source: §3.1 · p. 4

3.2 SPATIOTEMPORAL EVIDENCE MEMORY

  • The memory anchors retained RGB keyframes and acquisition context, preserving available temporal, depth, pose, calibration, and synchronization records for later processing.Initial fields include descriptions and features; subsequent tasks can add OCR, event, or geometric fields and query indices by reprocessing retained observations.Source: §3.2 · p. 4

3.3 ASYMMETRIC ACCESS AND DUAL-AGENT PROBLEM SOLVING

  • The execution agent uses a database-like tool interface, while the coding agent explores the same memory in a programmable workspace with an independent compute budget.The execution agent progressively requests images after receiving compact retrieval results; the coding agent can inspect evidence, run programs, and acquire models or dependencies.Source: §3.3 · pp. 4–5
  • The booth-counting example shows executor replay exposing revisit-related counting confusion, prompting pose-and-time merging before a mounted tool returns both booths and supporting frames in one call.The coding agent first develops photo retrieval after text-based retrieval fails.Source: Figure 3 · p. 5

3.4 EXPLORE–DISTILL–MOUNT

  • EDM develops solutions through exploration, distills reusable operations and preprocessing into tools and memory builders, then mounts them for execution.Exploration uses observations and traces without task reference answers; executor failures help locate difficulties to address, and builders can be rerun when memory data change.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate base memory retrieval on GOAT-core, capability expansion on OC-NaVQA and FindingDory, and reuse on held-out episodes and real-robot patrol memories. Results cover retrieval success, task success and localization, and online tool-call costs.

4 EXPERIMENTS

  • The experiments assess initial memory fields and multi-channel retrieval, EDM capability expansion on two benchmarks, and reuse of frozen capabilities on new queries and memories.The benchmark suite includes GOAT-core, OC-NaVQA, and FindingDory, with additional real-robot patrol-memory experiments.Source: §4 · p. 6, §4.2 · p. 6, §4.3 · p. 6

4.1 EXPERIMENTAL SETUP

  • Within each OC-NaVQA and FindingDory comparison, the executor model and online decision-round cap remain fixed before and after capability mounting.Opus-5 coding-agent results serve as an exploration reference under a separate compute budget.Source: §4.1 · p. 6

4.2 BASE MEMORY DESIGN ON GOAT-CORE

  • On GOAT-core, direct retrieval achieves 90.8% SR@5 with Sol annotation and 89.8% with Qwen, above MoM’s visual-only 81.5%; the agent achieves 90.6%.The evaluation covers image, object, and language queries across four scenes, with a 1.5 m success threshold.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 CAPABILITY EXPANSION WITH EDM

  • EDM reduces OC-NaVQA mean position error from 33.10 m to 24.72 m and raises the fraction within 10 m from 19.7% to 36.6%.The object-position tool combines image segmentation, depth backprojection, and multi-view aggregation to return target coordinates with supporting frames.Source: §4.3 · p. 6, Table 2 · p. 8
  • FindingDory success rises from 38.3% to 63.8% on ReSight and from 38.9% to 66.6% on RAVEN after EDM mounting.The comparison uses each host’s fixed executor and a 15-decision-round budget across 596 queries from episodes 91–100.Source: §4.3 · p. 7, Table 3 · p. 8
  • ReSight’s executor gains most on multi-goal and temporal tasks, improving from 7.9% to 50.6% and from 41.0% to 82.9%, respectively.Distilled tools support interaction enumeration, revisit ordering, duration ranking, and clock-time alignment; Figure 4 illustrates before-mounting execution, generated programs, and after-mounting execution.Source: §4.3 · pp. 7–8, Figure 4 · p. 7
  • On FindingDory episodes 81–90 excluded from exploration, frozen EDM tools and skills raise success from 37.5% to 65.8%, a 28.3-point gain.The 597-query evaluation makes no further capability updates.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 REAL-ROBOT EXPERIMENTS

  • On real-robot patrol memories, mounting raises success from 12/30 to 30/30 and lowers mean online tool calls from 5.10 to 1.20.Frozen builders also reconstruct fields for a new floor’s memory, where patrol-length success rises from 0/5 to 5/5 and photo localization from 4/5 to 5/5.Source: §4.4 · pp. 8–9, Table 5 · p. 9

5 LIMITATIONS

The evaluation uses predefined query sets and task categories, while real-world deployments may bring new long-tail queries as users’ needs change. Experiments demonstrate reuse across held-out episodes and new robot memories, but only partially capture this evolving distribution.

5 LIMITATIONS

  • The evaluation uses predefined query sets and task categories to enable controlled comparisons of memory access and capability expansion.Source: §5 · p. 9
  • Real-world deployment can introduce new, long-tail queries as users’ needs change.Source: §5 · p. 9
  • The experiments only partially capture evolving task distributions; longer deployments and broader emerging queries remain future evaluation directions.Source: §5 · p. 9

6 CONCLUSION

ReSight combines revisitable visual evidence with extensible memory capabilities, separating bounded execution from programmatic exploration. Its interface supports retrieval, progressive inspection, and later reprocessing of retained observations for new needs.

6 CONCLUSION

  • ReSight retains revisitable visual evidence and expands memory processing through EDM, with bounded execution separated from programmatic exploration.The system distills reusable tools, skills, and memory builders; experiments report memory access, improved question answering, and reuse across new queries and robot memories.Source: §6 · p. 9

A.1 BASE TOOL INTERFACE

  • Base retrieval returns text summaries, while position recall requires camera poses; object localization within an image and interaction-sequence recovery require additional operations.EDM adds tools for those operations.Source: Table 6 · p. 14, §A.1 · p. 14

A.2 RETRIEVAL FUSION AND PROGRESSIVE DISCLOSURE

  • Retrieval ranks visual, scene-description, and object-text channels independently, can filter by time or group nearby viewpoints, and opens original images only for verification.Compact text returns frame identifiers, descriptions, object fields, and available spatial and temporal metadata before image inspection.Source: §A.2 · p. 14

A.3 KEYFRAME FIELDS AND AN ANNOTATION EXAMPLE

  • Each retained keyframe links an image to capture time and available pose and depth metadata, while semantic text fields are encoded separately and linked by frame id.Later builders can reprocess retained images for OCR, interaction histories, or geometry without collecting observations again.Source: §A.3 · pp. 14–15

A.4 TWO WAYS TO ACCESS THE SAME MEMORY

  • The execution agent uses registered tools and skills within an online round limit, while the coding agent uses a workspace to inspect observations, write programs, and prepare models or dependencies.EDM adds reusable artifacts to a capability library for subsequent execution.Source: §A.4 · p. 15

B EXPLORE–DISTILL–MOUNT IMPLEMENTATION

EDM diagnoses capability gaps from execution and exploration records, then turns exploration-derived operations into mounted tools, builders, and skills. The procedure preserves the executor’s model and online budget while allowing a separate exploration process to develop reusable capabilities.

B.1 DIAGNOSING CAPABILITY GAPS FROM EXECUTION

  • Capability-gap diagnosis combines execution difficulty flags and answer disagreements with a ledger of available operations, without reference answers or evaluation scores.A gap map groups flagged queries by cause and connects each gap to operations and exploration programs.Source: §B.1 · p. 15

B.2 EXPLORATION, DISTILLATION, AND MOUNTING

  • Explore answers memory queries and produces programs; Distill maps missing or partial operations to tools or builders, and Mount registers them with applicable skills and query access.Builders may process whole memories before queries, while query-dependent computation runs when a tool is called; the executor model and online budget stay fixed.Source: §B.2 · pp. 15–16

B.3 EDM PROCEDURE

  • One EDM round produces a mounted configuration under the same online budget, with optional replay and refinement before skill generation.The procedure identifies RUN, FLAG, and MOUNT as scripts and other steps as coding-agent sessions.Source: §B.3 · p. 16, §B.2 · p. 16

B.4 EDM PROMPTS

  • The prompts require evidence-backed exploration without reference answers, diagnosis of gaps, tools built only for identified remedies, optional replay-based revision, and a skill grounded in traces or explorer notes.Tools return evidence rather than answers, and builders handle whole-memory computation while tools handle query-time computation.Source: §B.4 · pp. 17–18

C.1 COMPARISON PRINCIPLES AND REPRODUCED CONFIGURATIONS

  • Within a host, before-and-after comparisons hold the execution model, base interface, decision-round cap, input, and scoring rule fixed; cross-host comparisons retain each baseline’s memory representation and loop.The coding-agent exploration reference uses a separate workspace and compute budget.Source: §C.1 · p. 18

C.2 GOAT-CORE

  • GOAT-core evaluates 480 image, object, and language queries across four scenes and 1,606 keyframes, with localization success defined within 1.5 m on the horizontal XZ plane.Retrieval-only and agent configurations share a localization backend; the main table reports SR@5.Source: §C.2 · p. 19

C.3 OC-NAVQA AND FINDINGDORY

The evaluations cover OC-NaVQA and FindingDory, with capability additions assessed under specified host budgets and memory settings. Reported results include localization gains, held-out performance, and an exploration-free distillation comparison.

C.3 OC-NAVQA AND FINDINGDORY

  • OC-NaVQA contains 210 questions across seven sequences; position evaluation compares object-coordinate ground truth with camera-position returns before EDM and an object-coordinate estimate from the mounted tool.The reported metrics include mean position error, the share within 10 m, and temporal MAE over timestamp and duration questions.Source: §C.3 · p. 19
  • Localization improves with both annotators, while Sol annotations yield unchanged text accuracy and lower temporal error, and Qwen annotations yield unchanged text accuracy and higher temporal error.With Sol annotations, EDM corrects two of 67 binary answers and introduces five errors; binary accuracy remains unchanged with Qwen annotations.Source: §C.3 · p. 19
  • FindingDory reports official high-level success over 596 queries from episodes 91–100, and the frozen-capability evaluation applies tools and skills from those episodes to 597 queries in episodes 81–90.The held-out evaluation uses the same execution model and API backend, a 15-round limit, and at most 30 images per request.Source: §C.3 · p. 20
  • 110 queries missed by the unmounted executor are answered by distillation without exploration, while 57 previously answered queries are missed; against full EDM, the counts are 28 and 144.This configuration submits after 4.1 rounds on average and uses 26.9k tokens per query.Source: §C.3 · p. 20

C.4 REAL PATROL MEMORY

The real-patrol evaluation tests six memory queries across two floors, comparing base execution with EDM under fixed prompts and images. Success is scored against task-specific evidence and distance criteria.

C.4 REAL PATROL MEMORY

  • The real-patrol evaluation comprises six tasks: Warmth Corner sign retrieval, farthest-point distance, booth counting and Room 805 retrieval on floor 8, plus patrol length and photo localization on floor 1.Each task runs independently five times with the same prompt and image input in both configurations.Source: §C.4 · pp. 20–21
  • Each task runs independently five times with identical prompts and images in both configurations, within ten execution rounds and a separate 400-round exploration limit.Frozen programs are reused and, where needed, rebuild indices on floor 1.Source: §C.4 · pp. 20–21
  • The episode-level results are organized by the benchmark’s low-level query categories, with query counts reported alongside success.The table covers FindingDory episodes 81–90.Source: Table 8 · p. 21
  • Scoring requires legible sign evidence, both booth instances, a farthest-point result within 1.5 m of 31.07 m, and patrol length within 10% of 111 m.The farthest-point frame has a ±3-frame tolerance; sign evidence has a one-frame tolerance.Source: §C.4 · p. 21
  • Photo localization accepts a coordinate or cited camera pose within 3 m of the reference, while booth-count answers must cite evidence for both instances.The photo is an independently captured phone image, and both configurations receive identical images.Source: §C.4 · p. 21

C.5 STORE ANNOTATOR REPLACEMENT

The annotator-replacement evaluation tests whether EDM’s improvements persist when the stored annotations change while the executor remains fixed. The supplied passages report improvement across all FindingDory categories but do not provide the table’s numerical values.

C.5 STORE ANNOTATOR REPLACEMENT

  • For OC-NaVQA and FindingDory, locally served Qwen3.8-27B-FP8 replaces the GPT-5.6-sol writing-time annotator, while keyframes, prompt, parser, retrieval channels, executor, and online limits remain fixed.Programs distilled on the Sol-annotated store are reused on the Qwen-annotated store; local annotation incurs no annotation API charge.Source: §C.5 · p. 21
  • On either annotated store, EDM improves all three FindingDory categories.Table 10 reports high-level success on 596 queries from episodes 91–100, with the GPT-5.6-sol executor in every row.Source: §C.5 · p. 21, Table 10 · p. 22
  • The OC-NaVQA annotator-replacement comparison covers 210 questions and keeps the GPT-5.6-sol executor fixed across rows.The table also reports mean online input-plus-output tokens per query.Source: Table 9 · p. 22

D COMPUTE COST AND ONLINE USAGE

ReSight constructs annotations and EDM capabilities offline, then measures online token and model-request costs. EDM reduces both costs on FindingDory, while the OC-NaVQA subset shows a slight increase for ReSight.

D.1 OFFLINE CONSTRUCTION AND CAPABILITY EXPANSION

  • Keyframe annotation, vector encoding, and EDM capability construction occur offline and can serve multiple queries.Figures 5 and 6 report online execution costs.Source: §D.1 · p. 22

D.2 ONLINE TOKENS PER QUERY

  • EDM reduces online tokens on both FindingDory hosts, while the ReSight executor uses slightly more tokens on the OC-NaVQA subset.GOAT-core retrieval-only makes no online model request during candidate selection; its agent averages 6.7k tokens per query.Source: §D.2 · p. 22

D.3 ONLINE MODEL REQUESTS PER QUERY

  • Both FindingDory hosts make fewer online model requests after mounting EDM; ReSight makes slightly more on the OC-NaVQA subset.The request counts include executor decisions, answer submission, and any additional finalization.Source: §D.3 · p. 22

E COMPLETE EXECUTION TRAJECTORIES

Paired trajectories illustrate how EDM changes retrieval and evidence use across simulated benchmarks and real-robot memories. The demonstrations also show retrieved memory locations guiding later physical actions, which are verified with fresh observations.

E COMPLETE EXECUTION TRAJECTORIES

  • The trajectories present paired executions before and after EDM for two OC-NaVQA queries, two FindingDory queries, and six real-robot task types.They show model requests, tool calls, selected returns, visual evidence, and final answers.Source: E COMPLETE EXECUTION TRAJECTORIES · pp. 22–23

E.1 OC-NAVQA

  • For OC-NaVQA, EDM uses several views to estimate the semi-truck’s position rather than returning camera pose, while a presence scan narrows candidate frames for scooter verification.Both executions cite frame 67 for the semi-truck; the base execution misses the scooters.Source: Figure 7 · p. 24, Figure 8 · p. 24

E.2 FINDINGDORY

  • For FindingDory, EDM cites post-placement evidence for interaction time and ranks indexed interactions by duration to identify the object with the longest rearrangement.The base execution cites a frame before placement for interaction time and estimates duration from retrieved frames for the rearrangement query.Source: Figure 9 · p. 25, Figure 10 · p. 25

E.3 REAL ROBOT PATROL MEMORY

  • In patrol memories, EDM finds the legible Warmth Corner sign, computes the farthest point from 648 stored poses, and identifies both phone booths where base execution answers zero.For held-out examples, both executions answer the room 805 and photo-localization queries correctly; EDM uses fewer requests in the room 805 case and fewer tool calls and model requests in the paired photo-localization run.Source: Figure 11 · p. 26, Figure 12 · p. 26, Figure 13 · p. 27
  • For held-out patrol length, EDM sums consecutive positions across 501 frames to obtain 111.0 m, rather than the base estimate of 92.7 m from route fragments.The plotted trajectory is reconstructed from stored poses.Source: Figure 15 · p. 28

E.4 FROM MEMORY QUERIES TO PHYSICAL EXECUTION

  • After memory queries locate the two booths or a photo-matched viewpoint, the robot navigates to those locations and uses fresh observations to verify booth occupancy or wait at the destination.In the booth demonstration, it visits both booths and reports that both are empty.Source: §E.4 · p. 29, Figure 17 · p. 29, Figure 18 · p. 29

Related papers

Loading 2610.33487v1…