TriHelper: Zero-Shot Object Navigation with Dynamic Assistance

• TriHelper applies specific enhancements to each failure case and reports state-of-the-art Zero-Shot ObjectNav performance on Habitat.Source: §I · p. 1
• The Collision Helper uses clustering based on agent trajectories and encountered obstacles to address collisions.Source: §I · p. 1
• The Exploration Helper uses a learning mechanism based on agent exploration behavior to address low exploration efficiency.Source: §I · pp. 1–2
• The Detection Helper uses vision-language models to identify target objects in RGB images and improve detection accuracy.Source: §I · p. 2

A. Object Navigation

• ObjectNav uses semantic priors to help robots locate objects during visual navigation in unknown environments.Source: A. Object Navigation · p. 2

B. Zero-Shot Object Navigation

• Zero-Shot ObjectNav methods include semantic mapping (SemExp[13]), CLIP/RGB-D instruction fusion (CoW[16]), LLM-guided navigation (LGX[18], LFG[19], ESC[20]), and pixel-level targets (PixNav[12]).Source: B. Zero-Shot Object Navigation · p. 2
• VLMs can extract semantic information directly from RGB images, while LLMs select navigation goal points.Source: B. Zero-Shot Object Navigation · p. 2

A. Problem definition

• The task starts agents at random locations with one of six object categories as the target; success requires stopping within 0.1 meters of the target.Source: A. Problem definition · p. 2

1) Semantic Map:

• The semantic map is a C × W × H tensor built from RGB-D images and agent pose, with channels for obstacles, explored areas, agent locations, object categories, and false targets.The object-category masks are mapped alongside the geometric point-cloud representation.Source: 1) Semantic Map · p. 2

2) Frontiers Map:

• The frontier map is derived from explored-area boundaries and the obstacle map, with small areas removed before the remaining frontier centers are calculated.These centers form the candidate frontiers for exploration.Source: 2) Frontiers Map · pp. 2–3
• Frontiers are prioritized by balancing expected utility against travel cost, with λCU adjusting the relative importance of those factors.SCU(Fi) = U(Fi) − λCU C(Fi).Source: 2) Frontiers Map · p. 3

3) Frontiers Select:

• For each frontier, an LLM estimates whether its associated objects and the target object coexist there, and the frontier with maximum P(Fi) is selected dynamically.The selected FLLM is used later in the global policy.Source: 3) Frontiers Select · p. 3

A. Overview

• Long-term goals are updated every 10 time steps, and VLM-based target navigation is activated when semantic segmentation detects the target object.The target-navigation process continues toward the final target unless that target is false.Source: A. Overview · pp. 3–4

1) Collision Helper:

• When collision is detected, the helper clusters the navigable map and selects the centroid of its largest connected region as a long-term goal.The paper defines collisions as repeated obstacle contact while trapped or a local planner’s inability to find a reachable path to the set goal.Source: 1) Collision Helper · p. 4

2) Exploration Helper:

• The exploration helper calculates distance between the agent and its long-term goal, which the global policy uses to decide whether to put the LLM to sleep.The method is motivated by prolonged failure of baseline [7] to update its LLM-selected long-term goal.Source: 2) Exploration Helper · p. 4

3) Detection Helper:

• A VLM performs binary verification of a segmented target using the image’s other objects and room information, then supplies its response to the global policy.A false target is masked and recorded so the agent can continue exploring.Source: 3) Detection Helper · p. 4, 1) Global Policy · p. 4

1) Global Policy:

• The global policy invokes collision assistance after insufficient forward movement or an infeasible local path, and exploration assistance when the goal remains unchanged and the agent is near it.A verified target becomes the final target point; a false detection is masked and recorded, and may become the final goal if no target is found after the detection threshold.Source: 1) Global Policy · p. 4
• The three helpers are dynamically used throughout exploration and navigation to complement one another in ObjectNav.Fig. 3 illustrates their collaboration within the global policy.Source: 1) Global Policy · p. 4, Figure 3 · p. 4

2) Local Policy:

• FMM plans a path from the agent’s current location to the long-term goal at each timestep and selects an action without training.It replaces end-to-end methods such as reinforcement learning in the Zero-Shot framework.Source: 2) Local Policy · p. 5

A. Datasets

• The evaluation uses HM3D’s 2,000 episodes across 20 scenes and Gibson’s ObjectNav validation segment across five scenes, with six target categories in both datasets.The categories are couch, chair, bed, toilet, TV monitor, and potted plant.Source: A. Datasets · p. 5

B. Experiment Details

• The Habitat evaluation uses 480×640 RGB-D images, a base odometer, RedNet and YOLOv8 segmentation, and quantized Qwen-VL-Chat-Int4 for detection.The LLM for frontier selection is calibrated to the baseline framework; the dormant threshold is 50 cm, sleep time is 20, and detection threshold is 400.Source: B. Experiment Details · p. 5

C. Evaluation Metrics

• Success Rate (SR) measures task success, while Success weighted by Path Length (SPL) also evaluates path efficiency by comparing traveled and shortest path lengths.Higher SR and SPL are preferable.Source: C. Evaluation Metrics · p. 5
• Distance to Goal (DTG) measures the agent’s distance from the target at episode end, with lower values preferable.DTG uses the distance beyond the success threshold, bounded below by zero.Source: C. Evaluation Metrics · p. 5

D. Baselines

• FBE builds maps and selects frontiers, SemExp combines semantic mapping with reinforcement learning, and ZSON transfers an ImageNav-trained agent to ObjectNav.Source: D. Baselines · p. 5
• CoW uses CLIP and explores the nearest frontiers, ESC combines semantic mapping, CLIP, and LLM frontier selection, while L3MVN uses semantic maps and an LLM.PixNav trains pixel-goal navigation and uses LLMs for long-term planning; Randomly Walking selects actions randomly.Source: D. Baselines · p. 5
• PONI predicts two latent functions to locate unseen targets, Stubborn addresses collision and detection, and VLFM scores frontiers with a semantic value map.The comparison includes these and other Zero-Shot ObjectNav baselines.Source: D. Baselines · p. 5

E. Results

• Compared with the current state-of-the-art model, TriHelper improves SR by +4% on HM3D and +1.2% on Gibson.Source: E. Results · p. 5
• The authors attribute slightly lower SPL than the state-of-the-art method to the dynamic global policy’s SR focus and longer paths when false targets are blocked.They report that Gibson’s more spacious scenes and lack of upper- and lower-floor episodes diminish the collision-solving effectiveness.Source: E. Results · p. 6
• After manually removing simulator misjudgments in HM3D detection cases, the authors report a 5.9% improvement over baseline and 60%+ SR.Source: E. Results · p. 6
• TriHelper reaches state of the art across the reported baselines; versus the authors’ baseline it gains +6% SR and +9.5% SPL on HM3D, and +9.1% SR and +9.6% SPL on Gibson.Source: E. Results · p. 5, Table I · p. 6

F. Ablation study

• With all three helpers, the study reports 11.05% fewer collisions, 2.60% fewer false detections, and a 6% SR improvement over baseline.The individual helpers are reported to reduce collisions by 6%, failed exploration by 2.60%, and false detection by 3.05%, respectively.Source: F. Ablation study · p. 6, Table II · p. 6
• The combined setting also has increased exploration failures, especially when the target is on another floor; masking false targets can sometimes prevent target discovery before an episode ends.Source: F. Ablation study · p. 6
• Figure 4 presents Success Rate by category and method, enabling category-level comparison of the ablation results.Source: Figure 4 · p. 6

VI. CONCLUSIONS

• TriHelper combines Collision, Exploration, and Detection Helpers to address collision avoidance, exploration efficiency, and target detection in Zero-Shot ObjectNav.Source: §VI · p. 6
• The authors identify exploration across different floors as a problem for future research.Source: §VI · p. 6

A. Manually Double-check

• The authors manually double-checked HM3D detection cases for TriHelper and the baseline after finding some detection failures stemmed from simulator misjudgments.Source: A. Manually Double-check · p. 8
• After removing simulator-misjudgment cases, HM3D SR was 0.561 for the baseline and 0.620 for TriHelper, a 5.91% improvement.The table reports results of a manual double-check on HM3D.Source: A. Manually Double-check · p. 8, Table III · p. 8
• Simulator misjudgment cases accounted for 33.04% of baseline cases and 38.41% of TriHelper cases.Source: A. Manually Double-check · p. 8