PaperlayerView exact revision on arXiv

Source-linked AI summary

ApexNav: An Adaptive Exploration Strategy for Zero-Shot Object Navigation with Target-centric Semantic Fusion

Mingjie Zhang, Yuheng Du, Chengkai Wu, Jinni Zhou, Zhenchao Qi, Jun Ma, Boyu Zhou

arXiv:2504.14478v3cs.RO

TL;DR

Zero-shot ObjectNav must find specified objects efficiently despite weak semantic cues and noisy or misleading detections. ApexNav adaptively switches between semantic and geometry-based exploration and uses target-centric fusion of multi-frame evidence. It achieves state-of-the-art results across three benchmarks, with reported gains on HM3Dv1 and HM3Dv2, and is validated in real-world experiments.

  • Problem

    Zero-shot ObjectNav remains challenged by inefficient semantic exploration under weak cues and unreliable target identification under noisy or ambiguous detections.

  • Method

    ApexNav adapts between semantic and geometry-based exploration and fuses multi-frame observations of targets and similar objects using context-aware confidence weighting.

  • Results

    ApexNav achieves state-of-the-art zero-shot performance on HM3Dv1, HM3Dv2, and MP3D, with relative SR/SPL gains of 5.5%/8.6% on HM3Dv1 and 19.8%/16.9% on HM3Dv2 over previous methods.

  • Takeaways & Limitations

    Ablations support adaptive switching and target-centric fusion, while real-world experiments validate ApexNav’s practicality in physical environments.

  • Takeaways & Limitations

    ApexNav assumes the target is visible and may explore inefficiently when small objects require finer inspection or when targets have weak semantic relevance to their surroundings.

Abstract

from arXiv · show

Navigating unknown environments to find a target object is a significant challenge. While semantic information is crucial for navigation, relying solely on it for decision-making may not always be efficient, especially in environments with weak semantic cues. Additionally, many methods are susceptible to misdetections, especially in environments with visually similar objects. To address these limitations, we propose ApexNav, a zero-shot object navigation framework that is both more efficient and reliable. For efficiency, ApexNav adaptively utilizes semantic information by analyzing its distribution in the environment, guiding exploration through semantic reasoning when cues are strong, and switching to geometry-based exploration when they are weak. For reliability, we propose a target-centric semantic fusion method that preserves long-term memory of the target and similar objects, enabling robust object identification even under noisy detections. We evaluate ApexNav on the HM3Dv1, HM3Dv2, and MP3D datasets, where it outperforms state-of-the-art methods in both SR and SPL metrics. Comprehensive ablation studies further demonstrate the effectiveness of each module. Furthermore, real-world experiments validate the practicality of ApexNav in physical environments. The code will be released at https://github.com/Robotics-STAR-Lab/ApexNav.

I. INTRODUCTION

ApexNav addresses two ZSON challenges: inefficient exploration when semantic cues are weak and unreliable target identification under noisy or ambiguous detections. It adaptively combines semantic and geometric exploration with target-centric multi-frame fusion, achieving state-of-the-art benchmark performance and real-world validation.

  • Motivation: Zero-shot ObjectNav methods offer strong generalization but still face efficiency and robustness challenges in unknown environments.Semantic guidance can be unavailable or misleading, while single-frame and max-confidence detections remain vulnerable to misdetections.
  • Contributions: ApexNav switches between semantic reasoning and geometry-based exploration according to the environment’s semantic distribution.Semantic mode guides the agent toward likely target regions, whereas geometry-based mode rapidly explores unknown areas; high-scoring frontiers are ordered with TSP.
  • Contributions: Target-centric semantic fusion preserves long-term memory of targets and similar objects through context-aware weighting of multi-frame observations.This design supports reliable identification despite noisy detectors and visually similar objects.
  • Evaluation: 5.5% and 8.6% relative improvements in SR and SPL on HM3Dv1, and 19.8% and 16.9% on HM3Dv2, establish state-of-the-art zero-shot performance over previous methods.The paper also evaluates MP3D, conducts ablations, and reports real-world experiments confirming practicality.
  • Relation to prior work: ApexNav builds on zero-shot ObjectNav research using LLMs and VLMs to improve efficiency and reliability without additional task training.Related methods include geometry-based, semantic-map, scene-graph, and value-map frontier exploration strategies.

III. PROBLEM FORMULATION

The task requires an agent with RGB-D and odometry sensing to reach a specified object efficiently in an unknown environment. ApexNav represents explored space, estimates semantic relevance, and integrates observations to decide between target-directed navigation and frontier exploration.

  • III. PROBLEM FORMULATION: ObjectNav requires reaching a specified object in an unknown environment while minimizing path length and satisfying a defined success distance.The agent uses an egocentric RGB-D camera and odometry for displacement and orientation relative to the start.
  • IV. METHODOLOGY: ApexNav constructs a frontier map and semantic score map, then uses target-centric fusion to determine whether a reliable target is present.A detected reliable target becomes the object waypoint; otherwise, adaptive exploration selects a frontier waypoint.
  • 1) Frontier Map:: The frontier map represents free, occupied, and unknown cells on a 2D probabilistic grid, with frontiers defined at free–unknown boundaries.Depth-derived point clouds are denoised, and frontier clusters are incrementally updated and approximated by their centers.
  • 2) Semantic Score Map:: The semantic score map measures relevance between observed environment regions and the target using BLIP-2 image–text cosine similarity.Higher scores indicate stronger target correlation.
  • 2) Semantic Score Map:: Room-associated objects augment text prompts to improve semantic prediction and enable earlier long-range predictions.Objects without strong room associations use prompts mentioning only the object, while room names are inferred by an LLM.
  • 2) Semantic Score Map:: Semantic scores are projected onto observed free grids and weighted by viewing-angle confidence, with optical-axis regions receiving higher confidence.Confidence decreases with angular offset from the optical axis.
  • 2) Semantic Score Map:: Semantic information is integrated across frames using a confidence-weighted average of current and previous grid scores.The updated confidence uses squared weighting to emphasize more confident predictions.

B. Adaptive Exploration Strategy

ApexNav switches between semantic and geometry-based exploration according to the strength and distribution of semantic cues, using path planning to visit promising frontiers efficiently.

  • 1) Mode-switching Criteria:: The agent enters semantic mode when frontier scores are uneven and geometry mode when semantic cues are limited or uniform.The max-to-mean ratio r and standard deviation σ characterize semantic unevenness; both must exceed thresholds for semantic exploration.
  • 1) Mode-switching Criteria:: Semantic exploration prioritizes high-scoring frontiers, while geometry exploration selects the nearest frontier to expand coverage.The geometry-based strategy is adopted because ObjectNav prioritizes locating the target rather than exhaustive exploration.
  • 1) Mode-switching Criteria:: In the illustrated toilet search, the agent alternates modes as semantic evidence changes and finds the target at step 156.It uses geometry mode when scores are similar, semantic mode after detecting washing machines or a likely bathroom, and geometry mode again when evidence weakens.
  • 2) Semantic-based Exploration:: ApexNav uses an Asymmetric TSP over high-score frontiers to optimize their visiting order and reduce redundant travel.Navigation distances are computed from A*-planned paths, and the LKH-Solver computes the path visiting all selected frontiers.

C. Target-centric Semantic Fusion

Target-centric semantic fusion maintains object-level, multi-label memory across observations and combines confidence with detection volume to improve target recognition under noisy detections.

  • C. Target-centric Semantic Fusion: ApexNav maintains long-term memory of the target and visually similar objects through context-aware, multi-frame semantic fusion.The method uses object-level clusters rather than individual grids as the fundamental fusion units.
  • C. Target-centric Semantic Fusion: LLM-generated room and object prompts augment target-related semantic predictions, while adaptive confidence thresholds account for object-size effects.Objects associated with rooms support earlier predictions at longer ranges; objects without room-specific associations use object-only prompts.
  • 3) Semantic Label Confidence Fusion:: Each object cluster stores associated 2D grids and per-label 3D point clouds, confidence scores, and accumulated detection volumes.This multi-label structure preserves information for different object categories within the same cluster.
  • 3) Semantic Label Confidence Fusion:: Single-frame detections are projected onto the 2D map and incrementally associated with existing clusters or used to create new clusters.Unoccupied projected grid cells are filtered before intersection-based clustering and fusion.
  • 3) Semantic Label Confidence Fusion:: Fusion weights observations by detection volume, approximated by the number of points in a consistently down-sampled point cloud.Larger detected areas are treated as more reliable, and current observations are merged with stored point clouds.
  • 3) Semantic Label Confidence Fusion:: Missing current detections reduce confidence for corresponding object clusters, mitigating abnormal confidence from occasional false positives.The penalty is applied by assigning zero confidence to the missing detection and fusing it with volume-dependent weighting.
  • 3) Semantic Label Confidence Fusion:: The best-matching label is selected by maximizing confidence multiplied by detection volume, balancing reliability with observation count.This selection is performed after each cluster has accumulated information for multiple labels.

4) Selection of Reliable Targets:

ApexNav distinguishes reliable from suspected target clusters using a target-specific fused-confidence threshold and prioritizes reliable targets for navigation.

  • 4) Selection of Reliable Targets:: Clusters matching the target and exceeding cth are treated as reliable targets, while lower-confidence clusters remain suspected targets.Suspected targets are considered only after no frontiers remain, with the most confident one selected as the goal.

D. Safe Waypoint Navigation

ApexNav selects a frontier or object waypoint, then evaluates actions using distance and proximity costs to balance progress toward the local target with route quality.

  • ApexNav selects either a frontier or an object waypoint as the navigation goal, depending on whether a reliable target is present.
  • The agent evaluates discrete actions jointly for efficiency and safety because shortest paths can run too close to obstacles.
  • A* finds the shortest path, and the first point beyond dlocal becomes the local target plocal.
  • The distance cost costtarget encourages movement toward plocal.
  • The proximity change cost costprox penalizes actions that increase distance from plocal.

2) Safety Cost:

ApexNav adds safety-aware action selection and evaluates its approach across three Habitat datasets using SR, SPL, and SoftSPL.

  • Safety Cost: A local ESDF map samples K points along each action path to compute a safety cost.
  • Safety Cost: Only sampled points within dth of an obstacle contribute to costsafe, with ϵ preventing division by zero and ωs scaling the penalty.
  • Safety Cost: The lowest-total-cost action is selected, while MINCO generates safe continuous trajectories for real-world navigation.
  • Experimental Setup: Evaluation uses HM3Dv1, HM3Dv2, and MP3D, comprising 2000, 1000, and 2195 episodes respectively.
  • Experimental Setup: SR measures success, SPL measures efficiency relative to the optimal path, and SoftSPL captures goal-directed progress in failed episodes.
  • Comparison with State-of-the-art: ApexNav improves HM3Dv1 SR by 5.5% over TriHelper and HM3Dv1 SPL by 8.6% over VLFM.

C. Failure Cause Analysis

ApexNav failures arise from detection errors, annotation issues, cross-floor limitations, scene scale, and missing depth data; ablations support adaptive exploration.

  • Failure Categories: Failure categories include Success, Different Floor, False Positive, No Frontier, Stepout, and Missing Target.
  • Failure Causes: False Positive cases exceed 11% across datasets, mainly because persistent detection errors and incorrect fusion identify the wrong object.
  • Failure Causes: Annotation issues can cause failure even when the agent finds the target, because the object is not labeled.
  • Failure Causes: Different Floor failures exceed 13% in datasets other than HM3Dv2 because ApexNav builds only a single-layer 2D map.
  • Failure Causes: Starting near staircases can produce No Frontier failures when the 2D map finds no unexplored frontiers; 3D maps are proposed as a future remedy.
  • Ablation Study: Two-stage exploration outperforms single-stage strategies, with Greedy-Geometry effective under weak cues and TSP-Semantic effective after strong cues emerge.

2) Ablation Study on Exploration Metrics:

ApexNav’s ablations show that adaptive semantic–geometric exploration, target-centric fusion, safe waypoint navigation, and real-world deployment each address distinct efficiency, robustness, or practicality concerns.

  • Exploration balance: The best exploration balance occurs at σt = 0.015 and rt = 1.10, while relying too much on semantic or geometric information reduces efficiency.The results form a hump-shaped curve across SR, SPL, and SoftSPL.
  • Target-centric semantic fusion: Both w/o Fusion controls underperform Ours, while higher DCT reduces false detections but misses correct detections.ApexNav uses a lower DCT and refines more candidates through fusion, improving robustness to false detections.
  • Safe waypoint navigation: Removing safe waypoint navigation decreases SR and SPL and increases DTG compared with shortest-path navigation alone.The shortest-path condition often produces collisions that disrupt stable progress toward the target.
  • Real-world validation: Real-world experiments evaluate ApexNav on an AgileX LIMO robot across dormitory and laboratory environments.The system uses ROS communication and MINCO-generated collision-free spatiotemporal trajectories for waypoint navigation.
  • Overall framework: ApexNav combines semantic- and geometry-based exploration according to semantic-cue reliability and preserves memory of targets and similar objects.The conclusion reports state-of-the-art performance on three benchmark datasets and validation through real-world experiments.

APPENDIX

The appendix documents the LLM prompting and label-processing rules used to supply likely misdetections, confidence thresholds, and probable rooms for object detection.

  • Prompt design: The LLM is prompted to provide misdetection candidates, potential room objects, and a confidence threshold for each supplied label.The procedure uses system instructions followed by few-shot examples.
  • Output constraints: Misdetection labels must use 3–5 COCO classes, while confidence scores remain within [0.25, 0.65].The room is returned as “everywhere” when no room has substantially higher probability.
  • Few-shot example: For a cake, the example output lists donut, pizza, sandwich, and pie as similar objects, with threshold 0.30 and room “everywhere.”The examples justify these candidates using visual similarities such as circular shape, layers, and icing-like decoration.
  • Few-shot example: For a cabinet, the example returns bookshelf, dresser, and closet, with threshold 0.50 and kitchen as the most likely room.The prompt also acknowledges bathroom, hallway, and living-room occurrences depending on layout.
  • Dataset adjustments: Dataset preprocessing replaces selected labels and uses different confidence ranges for HM3D and MP3D detection difficulty.HM3D uses [0.25, 0.65], whereas harder-to-detect MP3D objects use [0.25, 0.35].

B. Details of Failure Cause Analysis

Failure analysis identifies detector misdetections and dataset labeling errors as remaining sources of task failure, even with target-centric semantic fusion.

  • Detection errors: Detector limitations can still misclassify incorrect objects as task targets, causing task failure despite target-centric semantic fusion.The failure is attributed to the performance limitations of the underlying detectors.
  • Dataset labeling errors: HM3D-v0.1 contains objects whose labels conflict with human recognition because they were not annotated as target objects.Many of these labeling issues were addressed in HM3D-v0.2.

C. Details of Ablation Experiments

Ablation analyses connect performance losses to missing semantic fusion and similar-object detection, with false positives and unstable target selection explaining representative failures.

  • Fusion ablation: Removing fusion causes a very significant SR decrease across all three datasets, with HM3D-v0.2 showing the greatest decrease.The w/o fusion group exhibits more “False Positive” and “Stepout” cases.
  • False Positive failures: Without fusion, partial views of non-target objects can trigger a mistaken STOP signal because current detections override prior observations.This produces “False Positive” failures.
  • Stepout failures: Without fusion, fluctuating highest-confidence detections cause left-right turning and prevent stable navigation-target selection.The resulting excessive steps produce “Stepout” failures.
  • Similar-object ablation: Removing similar-object detection significantly lowers SR on MP3D when ambiguous targets increase confusion with visually similar objects.Cabinets are cited as an example, and auxiliary similar-object detection helps correct these false positives.
Loading 2504.14478v3…