PaperlayerView exact revision on arXiv

Source-linked AI summary

How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers

Junting Chen, Guohao Li, Suryansh Kumar, Bernard Ghanem, Fisher Yu

arXiv:2305.16925v1cs.CVcs.LGcs.RO

TL;DR

Object goal navigation asks an agent to find a target object category in an unknown environment, while existing approaches can require costly training and be difficult to explain. StructNav uses visual SLAM, structured scene representations, and semantic frontier scoring in a training-free pipeline. The paper reports state-of-the-art performance on Gibson compared with previous training-heavy methods, while identifying the semantic model as a potential bottleneck.

  • Problem

    ObjectNav requires searching an unknown environment for a target category instance, while existing learning-based approaches can have low sample efficiency, questionable generalization, and limited explainability.

    Source: §I · p. 1

  • Method

    StructNav builds a scene representation with visual SLAM and selects promising geometric frontiers using semantic priors from language and scene statistics.

    Source: §I · p. 1, D. Exploration with Semantic Frontiers · p. 5

  • Results

    The training-free pipeline achieves state-of-the-art performance on Gibson with a large margin over previous training-heavy methods.

    Source: §V · p. 8

  • Takeaways & Limitations

    Pipeline ablations identify the semantic model as a major bottleneck for object goal navigation.

    Source: §V · p. 8

    The semantic model could be a major bottleneck in the pipeline.

    Source: §V · p. 8

Abstract

from arXiv · show

Object goal navigation is an important problem in Embodied AI that involves guiding the agent to navigate to an instance of the object category in an unknown environment -- typically an indoor scene. Unfortunately, current state-of-the-art methods for this problem rely heavily on data-driven approaches, \eg, end-to-end reinforcement learning, imitation learning, and others. Moreover, such methods are typically costly to train and difficult to debug, leading to a lack of transferability and explainability. Inspired by recent successes in combining classical and learning methods, we present a modular and training-free solution, which embraces more classic approaches, to tackle the object goal navigation problem. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. We then inject semantics into geometric-based frontier exploration to reason about promising areas to search for a goal object. Our structured scene representation comprises a 2D occupancy map, semantic point cloud, and spatial scene graph. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers. With injected semantic priors, the agent can reason about the most promising frontier to explore. The proposed pipeline shows strong experimental performance for object goal navigation on the Gibson benchmark dataset, outperforming the previous state-of-the-art. We also perform comprehensive ablation studies to identify the current bottleneck in the object navigation task.

I. INTRODUCTION

Object goal navigation requires an agent to search an unknown environment for an instance of a specified object category, combining scene understanding, exploration, planning, and collision avoidance. StructNav proposes a modular, training-free approach, while the related work describes trade-offs in learning-based alternatives and motivates evaluation beyond idealized localization.

I. INTRODUCTION

  • ObjectNav requires searching an unknown environment for a target category instance and combining scene understanding, exploration, path planning, and collision avoidance.The target’s position is not provided, so the agent must reason about unexplored areas using observations and scene information.Source: §I · p. 1
  • StructNav combines visual SLAM, a structured scene representation, and semantic frontier scoring to select promising unexplored areas without training an exploration policy.Its priors use language knowledge and scene statistics, with scene priors drawn from the training split of 25 Gibson scenes.Source: §I · pp. 1–2
  • The paper proposes evaluating visual-odometry localization in place of ground-truth localization and testing how noise in different robotic pipeline modules affects ObjectNav.The authors motivate this setting by noting that many approaches use idealized ground-truth localization.Source: §I · p. 2
  • The proposed modular pipeline is described as state-of-the-art for ObjectNav and has a simple learning-free policy module intended for ROS-based robots.The paper also proposes a more comprehensive visual-navigation benchmark and examines how common robotic-system building blocks affect ObjectNav performance.Source: §I · p. 2

II. RELATED WORK

  • Related work spans end-to-end learning and hybrid classical-learning methods, with hybrid approaches often retaining reinforcement-learning semantic exploration that requires extensive training data.The cited discussion notes that SemExp requires 10 million frames to train its network [7].Source: §I · p. 1, §II · p. 3

III. METHODOLOGY

The methodology formalizes ObjectNav as reaching an instance of a named object category in an unknown environment. It also distinguishes an idealized ground-truth localization setting from evaluation using visual odometry.

III. METHODOLOGY

  • The methodology is organized around defining ObjectNav, describing pipeline data flow, constructing the structured scene representation, and calculating navigation goals and actions.Source: §III · p. 3

A. Problem Definition of Object Navigation

  • In the formal task, an agent starts at a randomly initialized pose in an unknown environment and must navigate to an instance of the specified goal-object category.The Habitat action space includes turning left or right, moving forward, and stopping; observations come from an RGBD camera.Source: A. Problem Definition of Object… · p. 3
  • The evaluation replaces ground-truth relative pose information with localization from visual odometry to assess a more realistic setting.Source: A. Problem Definition of Object… · p. 3

B. Structured Navigation Pipeline

StructNav processes observations into a structured scene representation, then uses that representation to select navigation goals and actions. Its navigation stage combines geometric frontiers with semantic information from a spatial scene graph.

B. Structured Navigation Pipeline

  • StructNav updates its scene representation from observations, then exploits that representation to perform navigation actions.The pipeline runs in a loop between receiving sensor observations and sending actions to the agent controller.Source: B. Structured Navigation Pipeline · p. 3
  • The perception stage predicts semantic labels, estimates pose and reconstructs the scene with visual SLAM, then builds an occupancy map and semantic point cloud.The semantic segmentation image is predicted from the RGB image by a pretrained Mask R-CNN.Source: B. Structured Navigation Pipeline · p. 3
  • The navigation stage identifies boundary frontiers on the occupancy map, scores them using scene-graph semantics, and plans and executes actions toward the selected goal.Frontiers are cluster centers at the boundary between explored and unexplored map areas; global and local planners then guide the agent along the path.Source: B. Structured Navigation Pipeline · p. 3

C. Structured Scene Representation

StructNav represents the scene with an occupancy map, semantic point cloud, and spatial scene graph. It constructs these components on top of visual SLAM and uses clustering and filtering to form object nodes.

C. Structured Scene Representation

  • The structured representation contains a 2D occupancy map, a semantic point cloud with coordinates and labels, and a spatial scene graph of object nodes.The occupancy map is a slack bound of the explored area and is automatically expanded by the SLAM module.Source: C. Structured Scene Representation · p. 3
  • RTAB-Map builds the scene representation from RGBD observations, while DBSCAN clusters semantic points using label-weighted geometry to separate object categories.The graph’s object nodes represent object centers and labels, with edges representing translations between nodes.Source: C. Structured Scene Representation · pp. 3–4
  • Three-dimensional non-maximum suppression filters small clusters that overlap substantially with larger clusters to reduce faulty object nodes.The paper identifies false predictions and inter-frame inconsistency in 2D semantic images as sources of faulty 3D object nodes.Source: C. Structured Scene Representation · p. 4

D. Exploration with Semantic Frontiers

StructNav combines geometric frontier exploration with semantic utility from the spatial scene graph to select promising temporary goals. Category priors guide semantic scoring, while the policy begins geometrically and later relies on semantic utility.

D. Exploration with Semantic Frontiers

  • With sparse initial observations, geometric utility selects frontiers, favoring larger frontier lengths and shorter geodesic distances.The geodesic distance is measured on the 2D occupancy map.Source: D. Exploration with Semantic Frontiers · p. 4
  • Semantic frontiers combine scene-graph information and frontier geometry to select promising temporary navigation goals, using semantic priors to avoid unnecessary exploration.The utility module combines the two scores with prior matrices.Source: D. Exploration with Semantic Frontiers · p. 4, Figure 3 · p. 5
  • Semantic utility propagates nearby object-category relations to frontiers; prior distance and variance weight those relations, such as basin and toilet evidence for a bathtub.Relation scores use inverse prior distance, while discount weights use inverse square-root variance.Source: D. Exploration with Semantic Frontiers · p. 5
  • The category priors come from BERT and CLIP class-name embeddings and Gibson training scenes, with Gibson also supplying distance variances.The variance matrix weights prior distances.Source: D. Exploration with Semantic Frontiers · p. 5
  • The policy uses geometric utility for the first 50 steps, then semantic utility; object propagation uses either a frontier radius or a soft-radius distance weight.The soft-radius method adds inverse distance from an object to the frontier center as an extra weight.Source: D. Exploration with Semantic Frontiers · p. 5

A. Experiment Setup

Experiments use Gibson tiny in Habitat, following SemExp settings, and evaluate StructNav without training on the 25 training scenes. The setup also includes component-level analyses and reports success, path efficiency, and distance-to-goal metrics.

A. Experiment Setup

  • Evaluation uses Gibson tiny in Habitat with 25 training scenes, 5 test scenes, and 200 episodes per test scene.The simulated RGBD camera has 640x480 resolution, 79° field of view, and a 5-meter maximum depth range.Source: A. Experiment Setup · p. 5
  • The main comparison follows SemExp settings and tests whether StructNav achieves state-of-the-art ObjectNav performance without training on the 25 training scenes.The study also ablates semantic utility and supplies ground-truth data to pipeline components to analyze error accumulation.Source: A. Experiment Setup · pp. 5–6
  • StructNav first updates its structured representation from RGBD observations, then moves to the target if visible or navigates to a selected frontier otherwise.The pseudo-code describes representation updates using geometric and semantic information before navigation.Source: A. Experiment Setup · p. 6
  • The evaluation reports Success Rate, Success weighted by Path Length (SPL), and Distance to Goal (DTG).Success Rate is the share of episodes reaching the goal; DTG is average distance to the target at episode end.Source: A. Experiment Setup · p. 6

B. Implementation Details

RTAB-Map supplies pose and map outputs, while a separate instance reconstructs the semantic point cloud. Scene-graph extraction clusters semantic points into object instances and filters detections.

B. Implementation Details

  • RTAB-Map predicts agent pose and generates a 3D semantic reconstruction and 2D occupancy map.With external ground-truth odometry, only its map assembler and optimizer are used for point-cloud registration and global optimization.Source: B. Implementation Details · p. 6
  • Because RTAB-Map lacks native semantic SLAM support, a standalone instance reconstructs the semantic point cloud through 3D fusion.Source: B. Implementation Details · p. 6
  • Scene-graph extraction uses DBSCAN with a 1.0 neighbor distance and 5-point core threshold, then removes clusters under 5 points and applies 0.4-threshold box non-maximum suppression.Source: B. Implementation Details · p. 6

C. Object Goal Navigation on Gibson

StructNav achieves state-of-the-art ObjectNav results on Gibson, while ablations compare semantic utility variants and a qualitative episode illustrates target-directed frontier exploration. The authors also connect semantic reasoning and knowledge transfer to broader indoor robotic tasks.

C. Object Goal Navigation on Gibson

  • StructNav outperforms SemExp by 6.6% relatively on Success Rate, 22.7% relatively on SPL, and 6.9% relatively on DTG.The comparison used the same ground-truth-pose setting and shared semantic prediction model and local planner.Source: C. Object Goal Navigation on Gibson · p. 6, Table I · p. 7

D. Ablation Study on Semantic Utility Method

  • Ablations compare geometric-only exploration, CLIP and BERT semantic utility, and BERT variance, 3D-NMS, and soft-radius variants; geometric-only is competitive, while CLIP underperforms BERT.The authors suggest image-feature similarity may not correspond to spatial proximity between indoor categories.Source: D. Ablation Study on Semantic Utility… · p. 7

E. Qualitative Result of Semantic Utility Method

  • In the Wiconisco couch episode, SemUtil follows target-related frontiers to shortcut geometric exploration, which the authors connect to higher SPL.They suggest semantic reasoning and knowledge transfer may also help other indoor robotic tasks involving common-sense knowledge.Source: E. Qualitative Result of Semantic… · p. 7

F. Ablation Study on StructNav Pipeline

Pipeline ablations identify semantic segmentation as StructNav’s main bottleneck, while odometry also affects performance and simulation offers more tolerance to localization errors than real robots. The conclusion reports state-of-the-art Gibson results for the training-free pipeline.

F. Ablation Study on StructNav Pipeline

  • Pipeline ablations inject stepped ground truth for odometry, semantic segmentation, and scene graphs to measure module effects, with methods sharing a global planner.Ground-truth and predicted inputs use RTAB-Map or Mask R-CNN predictions and constructed scene graphs, respectively.Source: F. Ablation Study on StructNav Pipeline · p. 7, Table II · p. 7
  • Perfect semantic segmentation raises Success Rate about 15%, from 0.693 to 0.842, making segmentation StructNav’s major ablated bottleneck.The authors also report a degraded-segmentation ablation, with its results and discussion in Appendix VI-A.Source: F. Ablation Study on StructNav Pipeline · pp. 7–8
  • Ground-truth odometry raises Success Rate by 11.7% from 0.576 to 0.693 and SPL by 0.065 from 0.340 to 0.405.The discrete simulator action space and sliding at traversable-space boundaries give more tolerance to localization errors than real robotic systems.Source: F. Ablation Study on StructNav Pipeline · p. 8
  • Scene-graph extraction has little impact when segmentation is perfect; even with ground-truth odometry, segmentation, and graph, results remain 15.1% in Success Rate and 0.437 in SPL below Oracle.The authors attribute the success-rate gap to SLAM errors, especially in the projected 2D occupancy map, and the SPL gap more to scene-layout variance.Source: F. Ablation Study on StructNav Pipeline · p. 8

V. CONCLUSION

  • StructNav reaches state-of-the-art ObjectNav performance on Gibson with a large margin over previous training-heavy methods.The conclusion describes a training-free pipeline that builds a structured scene representation and uses observed partial scenes and category-to-category spatial priors.Source: §V · p. 8

VI. APPENDIX

This ablation tests how degraded semantic segmentation affects object goal navigation using artificial label noise and alternative corruption patterns. At 50% segmentation error, system performance is below half the ground-truth system's performance across all three metrics, while random label dropping is less damaging.

A. Ablation Study on Semantic Segmentation Models

  • The ablation compares ground-truth labels with class replacement at probabilities 0.1, 0.3, and 0.5, label dropping at 0.5, and Mask R-CNN.Artificial noise was added to ground-truth segmentation to assess the effect of degraded models.Source: A. Ablation Study on Semantic… · pp. 8–9
  • At 50% segmentation error, system performance is below 50% of ground-truth performance on Success, SPL, and DTG.The study identifies semantic segmentation quality as a potential bottleneck for object goal navigation.Source: Table III · p. 9, A. Ablation Study on Semantic… · p. 9, Figure 5 · p. 9
  • Randomly dropping labels has much less impact than other tested segmentation degradation; at 0.5 drop probability, recall is 0.5 while precision remains 1.0.The authors suggest temporal integration mitigates the effect: for N = 5, the probability of no label across all frames is 0.55 ≈ 0.031.Source: A. Ablation Study on Semantic… · p. 9
  • Figure 5 compares segmentation models and noise levels across Success, SPL, and DTG; higher Success and SPL are better, while higher DTG is worse.Source: Figure 5 · p. 9

Related papers

Loading 2305.16925v1…