AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigation

1 Introduction

• Zero-shot navigation targets specified objects in unseen environments without additional training, supporting deployment across diverse domains without environment-specific training or fine-tuning.The task uses a natural-language target category and visual information.Source: §1 · pp. 1–2
• Earlier ZSON methods often targeted single-floor environments, while MLLM waypoint-selection methods have struggled to generalize to multi-floor navigation.Larger environments increase demands on geometric perception and path planning.Source: §1 · p. 2
• Prior multi-floor methods may lack defined state transitions and adapt poorly to environmental variation, leaving robots vulnerable to abnormal situations and imbalanced exploration.Examples include failing to find a staircase entrance or getting stuck while reaching a frontier.Source: §1 · p. 2
• AERR-Nav proposes an Adaptive Exploration–Recovery–Reminiscing Strategy that dynamically switches among states to handle diverse navigation situations.Its Adaptive Exploration state uses fast thinking with semantic-value priors and uncertainty estimates, and slow thinking for commonsense-based high-level planning.Source: §1 · pp. 2–3
• AERR-Nav improves HM3D SR by +6.9% and SPL by +2.1%, and MP3D SR by +3.4% over the previously best-performing method.The introduction also reports state-of-the-art zero-shot performance on ZS-OGN benchmarks.Source: §1 · p. 3

2 Related Works

• Zero-shot navigation methods use pretrained VLMs or MLLMs for adaptability in unseen environments, but commonly use MLLMs as decision-makers without considering robot state.The paper describes an MLLM role-playing strategy in which actions depend on the robot’s current state.Source: §2 · p. 3
• Greedy semantic or MLLM-derived scoring can make exploration inefficient, while ApexNav and ASC have distinct limits in weak-cue coverage and environmental adaptation.ApexNav may overlook target-hidden areas with weak semantic cues; ASC often navigates to the nearest frontier.Source: §2 · p. 3
• AERR-Nav’s pipeline uses fast and slow thinking during exploration, distance-dependent recovery methods when entrapped, and keypoint-map revisiting after a floor is explored.The MLLM guides revisits to potential targets or stairway entrances.Source: Figure 1 · p. 4

3.2 Framework Overview

• AERR-Nav maintains persistent maps and switches among exploration, recovery, and reminiscing states as the environment changes.Exploration balances exploration and exploitation; recovery addresses getting stuck or wandering; reminiscing checks for missed targets and hidden stair entrances.Source: §3.2 · pp. 4–5

3.3 Map Representation

• The map representation combines visibility, exploration-value, and keypoint maps for each floor, with frontiers marking explored–unexplored boundaries.Frontier values combine semantic relevance and geodesic-distance preference; keypoints include room entrances and frontiers with open visibility.Source: §3.3 · pp. 5–6

3.4 State Detection and Switching

• The state machine starts in exploration while current-floor frontiers exist, enters recovery after a low-motion trigger, and enters reminiscing when no new frontiers remain.After recovery, the robot switches back to exploration.Source: §3.4 · p. 6

3.5 Adaptive Exploration State

• Slow-Thinking Mode uses room and object descriptions with MLLM reasoning to assess frontiers, with doorframes triggering shifts between thinking modes.RGB regions divided by doorframes are classified for object categories and room types before frontier suitability is evaluated.Source: §3.5 · p. 6, Slow-Thinking Mode · p. 8
• Fast-Thinking Mode selects intra-floor frontiers by balancing prior value with expected uncertainty reduction.The uncertainty estimate uses projected field-of-view coverage, semantic spatial density, and adjustments for overlap between frontiers.Source: Fast-Thinking Mode · p. 7
• The exploration reward adapts value and uncertainty weights, shifting from uncertainty reduction early toward value exploitation as mapping progresses and the step limit approaches.The reward incorporates unexplored-area share, discovered-frontier density, and temporal decay.Source: Fast-Thinking Mode · p. 7

3.6 Recovery State

• The dual recovery state handles stagnation at frontiers that are too near or distant, using path segmentation for distant targets and fine-grained MLLM actions for nearby ones.For distant frontiers, A* supplies a collision-free path and intermediate points; nearby-frontier actions integrate RGB images and the local map.Source: §3.6 · pp. 8–9, Figure 3 · p. 8

3.7 Reminiscing State

• The reminiscing state reviews keypoint RGB observations for missed targets, then searches for staircase entrances when no target location is identified.PointNav moves toward the keypoint most likely to contain a staircase, followed by MLLM-guided fine alignment.Source: §3.7 · p. 9

4.1 Experimental Setup

• HM3D validation includes 2,000 episodes across 20 scenes and six categories, while MP3D includes 2,195 episodes across 11 scenes and 21 categories.About 65% of HM3D scenes and 73% of MP3D scenes have multiple floors; evaluation uses SR and SPL.Source: §4.1 · p. 10

4.2 Comparison with SOTA

• AERR-Nav establishes state-of-the-art zero-shot performance on HM3D and MP3D, including gains over ASCENT and BeliefMapNav on the reported metrics.On HM3D, it achieves 72.3% SR and 35.6% SPL, improving over ASCENT by +6.9% SR and +2.1% SPL; on MP3D, it improves SR over ASCENT by +3.4% and SPL over BeliefMapNav by +1.1%.Source: §4.2 · p. 10, Table 1 · p. 11
• On HM3D inter-floor episodes, AERR-Nav outperforms ASCENT by +29.2% SR and +12.4% SPL, alongside steady intra-floor improvements.Episodes are divided according to whether the robot and target object start on the same or different floors.Source: Table 2 · p. 11, §4.2 · p. 11

4.3 Ablative Study

• REC raises SR by +1.2% on both datasets, while REM raises it by +1.6% on HM3D and +0.8% on MP3D; the full combination performs best.The study associates REC with rescuing agents from physical entrapments and REM with using keypoint memory to locate staircases and prevent target omission.Source: §4.3 · p. 12, Table 3 · p. 13
• GPT-4o has the highest efficacy among the tested MLLMs; BLIP-2 and CLIP achieve comparable results, both outperforming the BLIP baseline.BLIP-2 yields 72.3% SR and 35.6% SPL, while CLIP yields 72.1% SR and 35.5% SPL; the text reports GPT-4o as highest-efficacy without giving its scores here.Source: §4.3 · p. 12, Table 5 · p. 13, Table 6 · p. 13
• Semantic priors and uncertainty complement each other; dynamic weighting improves on fixed weighting, and MLLM reasoning improves navigation efficiency across tested configurations.With MLLM, visual reasoning can infer likely target areas rather than relying only on the exploration optimization scores.Source: §4.3 · p. 12, Table 4 · p. 13
• In a sofa-search case, AERR-Nav triggers MLLM reasoning at a detected door and explores the hallway, while ASCENT explores rooms described as unlikely to contain the sofa.The authors report that ASCENT's choice wastes steps, while AERR-Nav completes the task in fewer time steps.Source: §4.3 · pp. 12–13, Figure 5 · p. 14
• In a potted-plant cross-floor case, AERR-Nav recalls and locates a staircase entrance, while ASCENT wanders repeatedly and fails the task.The case description says MLLM hypothesizes a likely staircase location after the agent enters recall mode.Source: §4.3 · pp. 13–14, Figure 6 · p. 14

• AERR-Nav achieves state-of-the-art performance among zero-shot methods in experiments on HM3D and MP3D.Source: §5 · p. 14
• Its Adaptive Exploration–Recovery–Reminiscing strategy is designed to handle navigational contingencies.Source: §5 · p. 14
• Its Adaptive Exploration state lets the agent adapt dynamically to the environment.Source: §5 · p. 14