PaperlayerView exact revision on arXiv

Source-linked AI summary

RetroNav: Scene-Grounded Retrospection and Retry for Self-Evolving Lifelong Navigation

Diantao Tu, Yangdong Liu, Niannian Dai, Mengqi Rong, Shuhan Shen

arXiv:2610.35172v1cs.RO

TL;DR

Existing navigation methods often focus on isolated tasks, and lifelong approaches have tended to emphasize scene knowledge over historical decisions and their supporting evidence. RetroNav connects scene knowledge with navigation experience and uses post-attempt evaluation to guide evidence-grounded retries. It achieves state-of-the-art performance across five navigation benchmarks.

  • Problem

    Lifelong navigation methods have largely emphasized accumulated scene knowledge while neglecting historical interpretations and decisions linked to scene evidence.

    Source: §1 · p. 1

  • Method

    RetroNav links geometric and semantic scene knowledge with historical navigation experience, then uses independent post-attempt evaluation and retrospection to guide retries.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    RetroNav achieves state-of-the-art performance across five navigation benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Associating prior decisions with scene evidence supports revisiting navigation experience for knowledge reuse and failure reflection.

    Source: §1 · p. 1

    Memory construction and failure reflection depend on upstream perception, spatial association, and model judgment; retries also add travel steps and inference costs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Existing robot navigation methods primarily focus on isolated tasks, overlooking the need to accumulate knowledge and experience across repeated navigation episodes. Although recent lifelong navigation methods have explored cross-task knowledge accumulation, they mainly focus on scene knowledge while neglecting historical navigation experience and its connection to scene evidence, limiting the ability to reflect on past mistakes and improve future decisions. To address this limitation, this paper presents RetroNav, a training-free self-evolving navigation framework that integrates navigation, memory accumulation, and failure reflection. Specifically, we develop a unified memory system that connects geometric and semantic scene knowledge with historical navigation experience, distinguishing objective observations from task-dependent interpretations. After each navigation attempt, an independent evaluator determines whether a retry is necessary. If so, a Retrospection Agent revisits historical decisions using accumulated scene knowledge and navigation records, generating an evidence-grounded retry brief to guide the next attempt. RetroNav achieves state-of-the-art performance across five navigation benchmarks. Our method will be open-sourced at https://github.com/3dv-casia/RetroNav .

1. Introduction

Lifelong navigation needs to reuse both environmental knowledge and experience from prior decisions, because scene knowledge alone cannot reveal decision errors. RetroNav links scene evidence with navigation records and uses evaluation-driven retrospection to guide retries.

Motivation

  • Repeated navigation episodes can provide environmental information for future tasks, so systems should accumulate knowledge and experience beyond individual task completion.This applies to episodes whether they succeed or fail.Source: §1 · p. 1
  • Existing lifelong methods [11] [16] [19] [39] [46] mainly accumulate scene knowledge, while often overlooking how tasks were interpreted and decisions were made.A robot may observe the correct target but choose the wrong direction after misinterpreting an instruction.Source: §1 · p. 1
  • Effective lifelong navigation requires associating historical decisions with scene evidence so that experience can be revisited and reused.Scene knowledge captures reusable environmental structure, while navigation experience records how earlier tasks were interpreted and executed.Source: §1 · p. 1

Proposed framework

  • RetroNav combines navigation, memory accumulation, and failure reflection in a training-free framework with an evaluator-triggered, evidence-grounded retry loop.The Retrospection Agent reviews prior decisions using accumulated scene knowledge and navigation experience to produce a Retry Brief.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Prior work addresses persistent-scene navigation, scene-memory construction, and training-free agent navigation. RetroNav connects these areas by linking environmental evidence to task-dependent judgments and combining persistent memory with post-attempt reflection.

Lifelong navigation in persistent environments

  • Lifelong navigation methods reuse information from ordered tasks in persistent scenes, while RetroNav explicitly links task decisions to environmental evidence for inspection and targeted retries.Earlier methods use persistent scene knowledge, continual policy learning, and experience reuse.Source: §2 · pp. 2–3

Scene memory construction

  • Scene-memory approaches retain spatial, semantic, or visual structure; RetroNav keeps historical interpretations and actions as experience records rather than scene facts.Its memory links observations, poses, transitions, and views to regions, places, and objects.Source: §2 · p. 3

Training-free agent navigation

  • Training-free navigation methods use frozen foundation models, while RetroNav combines fixed model parameters with persistent scene memory, independent evaluation, and evidence-grounded retry reflection.The described prior approaches primarily focus on single-task control or local history.Source: §2 · p. 3

3. Method

RetroNav couples navigation, persistent scene-memory construction, and failure reflection across repeated tasks. Its unified memory connects scene evidence to task-specific decisions, while independent evaluation determines whether a failed attempt should be retried.

3.1. Problem Setting and Overview

  • Tasks specify a goal and start pose; retries address the same goal from its original start, while subsequent tasks introduce new goals and persistent memory remains within the evaluation scene and task-sequence boundaries.Scene knowledge accumulates across tasks, whereas a retry brief revises the decision context for another attempt at the same task.Source: §3.1 · p. 3

3.2. Unified Memory

  • Unified memory links cumulative scene knowledge with chronological navigation experiences through observation-centric associations, connecting shared scene evidence to task-specific judgments and actions.The associations support querying from decisions to accumulated scene evidence and from places or objects to nearby historical attempts and failures.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Scene knowledge records places, regions, persistent 3D objects, and verified traversability transitions; observations retain immutable empirical evidence while semantic classifications may be revised.Both successful and failed attempts contribute to scene consolidation.Source: §3.2 · pp. 3–4

3.3. Memory-Assisted Navigation

  • The View Selection Agent uses the goal, observations, local map, history, and retry brief to select a view or stop, querying scene context and relevant past decisions when needed.It records which retrieved experiences influenced each decision so later reflection can check whether they were applied correctly.Source: §3.3 · p. 5
  • The Waypoint Selection Agent converts semantic motion guidance into a reachable destination using depth-derived candidates, while focusing on local collision avoidance and traversability.Execution feedback and each observation, rationale, and action are appended to the attempt trace.Source: §3.3 · pp. 4–5

3.4. Evaluation and Retrospection

  • An independent evaluator checks landmark and instruction adherence and verifies the stopping position against updated scene memory, triggering a retry only when it judges failure.This avoids blind, unnecessary retries.Source: §3.4 · p. 5
  • After a failure, the Retrospection Agent checks for goal or landmark misidentification, route errors, and misapplied cross-task experience, then builds a retry brief.The brief can clarify the goal, review up to three pivotal mistakes, suggest a route, and include up to eight reference images.Source: §3.4 · pp. 5–6
  • The robot restarts from its initial pose with the brief and reference images, while still using real-time camera views for movement and collision avoidance.The brief acts as a checklist against repeating earlier mistakes.Source: §3.4 · p. 6

4. Experiments

Across episodic and persistent navigation benchmarks, RetroNav reports strong results, while ablations examine persistent memory, failure evaluation, and structured retries. A real-robot deployment also demonstrates autonomous object-goal navigation in a lobby.

4.1. Experimental Setup

  • Across five benchmarks, cross-task memory is disabled for episodic R2R-CE, RxR-CE, and HM3D-OVON, but enabled for persistent IR2R-CE and GOAT-Bench.Main results use representative subsets and average three independent runs.Source: §4.1 · pp. 6–7

4.2. Episodic Navigation

  • 84.0% SR on both R2R-CE and RxR-CE, plus 66.9% SR and 50.2% SPL on HM3D-OVON, accompany consistent gains over training-free methods.With GPT-6-Astra, RetroNav also exceeds AgenticNav on R2R-CE and ReflectVLN on RxR-CE; results extend across model families.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Navigation in Persistent Environments

  • IR2R-CE yields 89.2% SR and 78.8% t-nDTW; GOAT-Bench reaches 84.1% subtask SR and 40.3% e-SR, exceeding NavHarness’s 36.9% e-SR.The IR2R-CE result reflects destination success and adherence to demonstration paths across repeated visits.Source: Table 3 · p. 6, §4.3 · p. 7, Table 4 · p. 7

4.4. Ablation Study

  • Unified Memory raises IR2R-CE SR from 65.6% to 69.4% and t-nDTW from 68.6% to 74.2%; the full component configuration achieves 84.1% SR and 35.3% GOAT-Bench e-SR.The full configuration combines Unified Memory, the Evaluator, and Retry Brief; the authors attribute its gains to grounded causal retrospection rather than extra execution budget.Source: Table 5 · p. 8, §4.4 · p. 8
  • 25.8% recovery with the complete Retry Brief exceeds blind retry’s 12.1% by 13.6 percentage points on 33 first-attempt failures.Decision reviews identify choices to correct, while reference images provide cues for carrying out corrections.Source: Table 6 · p. 8, §4.4 · pp. 8–9
  • The Evaluator achieves 65.0% recall on failed trajectories and 71.1% overall accuracy, compared with 0.0% failure recall for navigator self-stopping.It does not use privileged simulator state.Source: Table 7 · p. 9, §4.4 · p. 9

4.5. Real-World Deployment

  • A CASBOT 02 humanoid robot autonomously navigates from an initially empty scene memory to a vending machine in a real lobby.The task runs without human intervention, using a chest-mounted RGB-D camera and a closed loop of observations, decisions, and motion execution.Source: §4.5 · pp. 9–10, Figure 5 · p. 9

5. Conclusion

RetroNav joins scene observations with navigation history so prior decisions can be revisited against accumulated evidence, and reports state-of-the-art performance across five benchmarks. Its memory construction and reflection depend on upstream perception, spatial association, and model judgment, while retries add travel and inference costs.

5. Conclusion

  • RetroNav links raw environmental observations with navigation experiences, enabling prior choices to be re-examined while keeping observations distinct from interpretations.After an independent evaluator judges an attempt unsuccessful, reflection reviews prior decisions against accumulated scene evidence and produces a retry brief.Source: §5 · p. 10
  • RetroNav achieves state-of-the-art performance across five navigation benchmarks by bridging cross-task knowledge reuse with within-task decision revision.Source: §5 · p. 10
  • Memory construction and failure reflection depend on upstream perception, spatial association, and model judgment; retries add travel steps and inference costs.The authors identify adaptive retry policies and more robust cross-task association under sensor noise as future work.Source: §5 · p. 10

Navigation

Table 8 summarizes evaluation subsets, cross-task memory settings, and navigation step budgets across five benchmarks.

Navigation

  • Table 8 summarizes evaluation subsets, cross-task memory settings, and navigation step budgets across five benchmarks.Source: Table 8 · p. 13

6. Evaluation Protocols

The evaluation uses benchmark-specific subsets and memory settings, with attempt-level navigation budgets that apply equally to retries.

Evaluation subsets

  • Evaluation subsets come from the corresponding val-unseen splits; IR2R-CE uses 299 episodes across 11 tours and three scenes, while GOAT-Bench uses 307 subtasks across five scenes.R2R-CE uses the 100-episode subset adopted by Open-Nav [31].Source: §6 · p. 13

Memory accumulation

  • Cross-task memory is disabled for R2R-CE, RxR-CE, and HM3D-OVON, but accumulates within scenes across successive tasks for IR2R-CE and GOAT-Bench.Each scene has separate memory; completed attempts update it before later tasks use it.Source: §6 · p. 13

Navigation budgets

  • Step limits count decision steps per attempt, and each retry restarts at the original pose with the same budget as the first attempt.An attempt ends when the navigator issues STOP or reaches its step limit.Source: §6 · p. 13

7. Memory Storage, Updates, and Queries

RetroNav’s memory lifecycle connects scene structure and episodic decisions, supports bounded retrieval during navigation and reflection, and accumulates spatial coverage and detail across trajectories.

7. Memory Storage, Updates, and Queries

  • The section traces the unified-memory lifecycle from record representation through updates and queries to multi-trajectory spatial accumulation.The schemas and operational implementation are presented as the basis for this lifecycle.Source: §7 · p. 13, Table 9 · p. 13

7.1. Record Layout and Evidence References

  • The memory schema separates persistent scene knowledge from episodic decision traces and links them through provenance and contextual-attribution indexes.Scene records cover places, regions, objects, and verified transitions; decision traces retain goals, observations, choices, rationales, and outcomes.Source: §7.1 · pp. 13–14

7.2. Incremental Update Pipeline

  • Each completed attempt updates unified memory through object evidence extraction, 3D association, scene organization, and links to navigation experience.Object annotation and segmentation precede depth back-projection and fusion with prior object instances; the association score selects the highest-scoring candidate, fusing above 0.5 and otherwise adding a new instance.Source: §7.2 · pp. 14–15
  • Scene updates retrieve nearby historical context, revise places and regions, record executed traversability, and preserve historical decisions through observation-anchored links and validation.The committed memory includes scene knowledge, navigation experiences, associations, and the updated object map.Source: §7.2 · p. 15

7.3. Query Interfaces and Context Budgets

  • Online planning retrieves bounded local scene context and, when needed, historical decisions relevant to current places, entities, or subgoals.The local window is capped at eight places; cross-task retrieval returns up to three event summaries and two detailed decision fragments.Source: §7.3 · p. 15
  • Online navigation accesses only memory committed before the active attempt, whereas reflection can inspect newly committed evidence from a failed run.Reflection search returns up to five comprehensive records and can inspect historical and newly committed images.Source: §7.3 · pp. 15–16

7.4. Spatial Expansion and Place-Level Enrichment

  • Memory expands into unseen areas while enriching known places without spurious region proliferation.In R2R-CE, regions and places grow from 2 and 5 to 12 and 27 after 28 trajectories; between trajectories 5 and 10, regions remain at 11 while places rise from 15 to 23.Source: §7.4 · p. 16
  • GOAT-Bench likewise grows from 5 places in one region to 9 regions and 32 places, while stationary or failed attempts add instance evidence without redundant trajectory edges.Source: §7.4 · pp. 16–17

8. Additional Experimental Results

Repeated tours improve navigation performance while reflective retries increase task completion across benchmarks. The section also reports the efficiency trade-offs of cumulative retry costs and joint view-waypoint selection.

8.1. Navigation Benefits over Repeated Tours

  • Across three consecutive tours of 82 routes, final SR climbs from 82.5% to 88.2% and SPL from 68.0% to 76.6%, while mean cumulative steps decrease by 11.7%.Memory accumulates across tours without resets; the reported explanation is reuse of confirmed landmarks and avoidance of previously explored dead ends.Source: §8.1 · p. 17

8.3. Joint View and Waypoint Selection

  • On R2R-CE, joint view-waypoint selection underperforms two-stage selection across navigation metrics, despite lower latency, and consumes 390.92 K vs. 265.41 K tokens.The joint scheme uses about one extra decision step; its additional late-episode context increases cumulative input tokens despite fewer model calls per decision.Source: Table 12 · p. 18, §8.3 · p. 18

8.2. Multi-Attempt Recovery and Efficiency

  • On IR2R-CE, reflective retry raises GPT-6-Astra SR from 80.0% to 89.2% and GPT-5.6-Sol from 72.4% to 84.1%; on R2R-CE and HM3D-OVON, GPT-6-Astra SR also rises.On R2R-CE, GPT-6-Astra increases from 77.3% to 84.0%; on HM3D-OVON, it increases from 61.8% to 66.9%.Source: §8.2 · pp. 17–18
  • Cumulative SPL counts distance across every attempt: GPT-6-Astra reaches 68.5% on IR2R-CE and 55.2% on GOAT-Bench, versus 58.4% final-attempt SPL on GOAT-Bench.Tasks without retries retain standard SPL; the simulator resets the agent between attempts without adding a traveled segment.Source: §8.2 · p. 18
  • With higher first-attempt accuracy and reduced exploration detours, GPT-6-Astra requires significantly fewer model calls and tokens, reducing cumulative navigation latency across persistent sessions; inference accounting includes all attempts and tool rounds.Background object annotation remains decoupled and is handled by GPT-5.6-Terra at approximately 482 s per episode on IR2R-CE.Source: §8.2 · p. 18
Loading 2610.35172v1…