PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, Wolfram Burgard

arXiv:2403.17846v2cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Large-scale open-vocabulary maps remain difficult to represent efficiently and to query beyond individual objects. HOV-SG builds hierarchical 3D scene graphs over floors, rooms, and objects, enabling abstract-query retrieval and multi-floor navigation; it reports higher semantic accuracy and a 75% smaller representation than dense maps.

  • Problem

    Large-scale environments and abstract queries beyond the object level remain difficult for open-vocabulary maps, while scalable representations must support hierarchy, efficiency, actionability, and open-vocabulary querying.

  • Method

    HOV-SG constructs multi-story 3D scene-graph hierarchies whose floor, room, and object nodes carry open-vocabulary features and support hierarchical query scoring and navigation.

  • Results

    HOV-SG surpasses previous baselines in open-vocabulary semantic accuracy at object, room, and floor levels while reducing representation size by 75% compared with dense open-vocabulary maps.

  • Takeaways & Limitations

    The representation supports concept retrieval from abstract language queries and long-horizon robotic navigation across real-world multi-story indoor environments.

Abstract

from arXiv · show

Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.

I. INTRODUCTION

HOV-SG addresses the difficulty of representing large-scale scenes and abstract language queries with scalable, hierarchical, open-vocabulary 3D scene graphs. It combines floor, room, and object concepts with language-grounded navigation capabilities.

  • Large-scale open-vocabulary scene representations must provide object-centric hierarchies, storage efficiency, actionability, and true open-vocabulary querying.
  • HOV-SG indexes floors, rooms, and objects with open-vocabulary features in hierarchical 3D scene graphs spanning multi-story environments.
  • HOV-SG decomposes abstract queries across hierarchy levels and supports object retrieval and long-horizon navigation in large-scale indoor environments.
  • The work evaluates semantic segmentation on Replica and ScanNet, analyzes scene-graph properties on Habitat-Matterport, and conducts real-world multi-floor navigation experiments.
  • Existing semantic maps often use fixed category sets, while open-vocabulary feature maps can incur substantial storage overhead by assigning embeddings to geometric elements.

C. Scene Graphs for Planning

HOV-SG constructs compact, hierarchical scene graphs from open-vocabulary 3D segments and connects them to planning over multi-floor environments. Its pipeline fuses visual-language features, merges segments, and organizes scenes into floors and rooms for query-based navigation.

  • Scene Graphs for Planning: HOV-SG targets indexing of floor-, room-, and object-level concepts from natural-language queries while supporting semantic and spatial robot navigation.
  • 3D Segment-Level Open-Vocabulary Mapping: The segment-level map assigns each 3D segment an open-vocabulary feature rather than storing an independent visual-language feature for every 3D point.
  • 3D Segment-Level Open-Vocabulary Mapping: Segments are merged through a fully connected overlap graph, where highly connected subgraphs are subsequently combined.
  • 3D Scene Graph Construction: Floor boundaries come from filtered peaks in a height-density histogram, while room masks are extracted with the Watershed algorithm.
  • 3D Segment-Level Open-Vocabulary Mapping: The feature pipeline fuses CLIP embeddings from the full frame, a mask crop, and a background-excluded mask crop using a weighted sum.
  • 3D Segment-Level Open-Vocabulary Mapping: The fused mask features are projected into global 3D coordinates, associated with nearby reference points, averaged per point, and used to infer features for merged 3D segments.

B. 3D Scene Graph Construction

HOV-SG constructs a hierarchical graph linking floors, rooms, and objects, enriches nodes with open-vocabulary features, and adds floor-level and cross-floor Voronoi navigation graphs.

  • Hierarchical graph: HOV-SG formalizes nodes for the scene root, floors, rooms, and objects, with hierarchical edges connecting these concept levels.Each non-root node stores its concept point cloud and associated open-vocabulary features.
  • Room features: Room nodes receive representative CLIP view embeddings from RGB-D observations whose camera poses lie inside the room segment.K-means distills the observations into k representative embeddings, which support room-category voting through cosine similarity.
  • Object association: Object segments are assigned to rooms using bird’s-eye-view point-cloud overlap, with nearest-room assignment when no overlap exists.This associates object-level geometry with the room hierarchy.
  • Navigational graphs: Single-floor navigation constructs a navigable area by removing obstacles from camera poses and floor point-cloud projections, then builds a Voronoi graph.The resulting graph represents traversable structure within each floor.
  • Navigational graphs: Cross-floor navigation extracts camera positions in stair regions and connects this subgraph to the corresponding floor-level Voronoi graphs.This extends actionable navigation across multiple floors.

C. Navigation with Scene Graph

HOV-SG grounds abstract language instructions by decomposing them across floor, room, and object levels, then uses hierarchical scoring and a navigational graph to reach the selected target.

  • Query decomposition: HOV-SG decomposes complex instructions such as “find the toilet in the bathroom on floor 2” into floor-, room-, and object-level queries.This extends navigation goals beyond isolated objects to spatial concepts such as rooms and floors.
  • Hierarchical grounding: The system sequentially scores query concepts against floors, rooms, and objects to progressively narrow the solution corridor.Cosine similarity identifies a target node at each hierarchy level.
  • Path planning: After target identification, HOV-SG uses its navigational graph to plan a path from the starting pose to the target destination.The graph supports actionable grounding of decomposed language queries.
  • Evaluation: The experiments examine semantic segmentation, scene-graph accuracy, large-scale real-world language-grounded navigation, and compact memory usage.These goals span ScanNet, Replica, and Habitat Matterport 3D Semantics evaluations.

A. 3D Semantic Segmentation on ScanNet and Replica

HOV-SG is evaluated for open-vocabulary 3D semantic segmentation and scene-graph structure, with qualitative Replica examples showing correct door boundaries and sofa predictions.

  • Benchmark evaluation: HOV-SG is compared with ConceptFusion and ConceptGraphs on open-vocabulary 3D semantic segmentation using ScanNet and Replica scenes.The evaluation uses different CLIP backbones and reports results in Table I.
  • Benchmark evaluation: HOV-SG outperforms the open-vocabulary baselines by a large margin on both mIOU and F-mIOU.Predicted object labels are projected to ground-truth points through five-nearest-neighbor majority voting for evaluation.
  • Qualitative results: On Replica qualitative examples, only HOV-SG predicts the door with correct boundaries at position A and the sofa correctly at position B.The figure legend displays ten of the dataset’s 101 categories.
  • Scene-graph structure: 100% accuracy is achieved for retrieving the number of floors in both single-floor and multi-floor HM3DSem scenes.Region segmentation recall reaches 83.59%, compared with 77.55% for Hydra, while precision is slightly lower.
  • Room semantics: The room-classification evaluation compares HOV-SG’s view-embedding method with GPT-3.5 and GPT-4 baselines under privileged and unprivileged settings.Exact text equality and human-assessed semantic correctness are evaluated separately.

3) Object-Level Semantics:

The object-level evaluation introduces AUCtop-k for large open-vocabulary label sets and compares HOV-SG with VLMaps and ConceptGraphs on Habitat-Semantics.

  • Metric: AUCtop-k measures the area under the top-k accuracy curve between predicted and ground-truth categories across the 1624-category label set.It reflects how many ranked candidate labels are needed on average before the ground-truth category appears.
  • Evaluation protocol: The evaluation uses linear assignment between predicted and ground-truth objects and retains predicted objects with IoU greater than 50%.Because VLMaps does not predict masks, it uses HOV-SG’s predicted object masks for comparison.
  • Results: HOV-SG achieves 84.88% AUCtop-k, compared with 84.07% for ConceptGraphs.HOV-SG outperforms ConceptGraphs through the top-100 highest-ranking classes.
  • Results: VLMaps identifies only 5% of objects correctly within its top-5 choices and reaches the correct class within top-500 predictions in 40.02% of cases.The passage attributes this weaker result presumably to dense feature aggregation and dependence on the fine-tuned LSeg model.

4) Hierarchical Concept Retrieval:

HOV-SG decomposes hierarchical language queries into floor, room, and object concepts, then retrieves objects by progressively querying the scene graph.

  • Query decomposition and retrieval: Hierarchical queries are decomposed into concepts such as [floor 2, living room, pillow] or [-, kitchen, bottle] before CLIP-based scoring.The system queries the most suitable floor, room, and object in sequence.
  • Retrieval results: 11.69% higher performance is reported for object-room-floor queries, while object-room queries gain 2.2% over the augmented ConceptGraphs baseline.ConceptGraphs struggles on larger scenes and more detailed queries.
  • Evaluation protocol: Top-5 retrieval accuracy is evaluated over 20 frequent object categories, with success requiring IoU > 0.1.The evaluated categories include objects such as pillows, doors, lamps, chairs, tables, towels, and plants.
  • Navigation evaluation: Navigation success is counted when the robot reaches within approximately 1 m of the sought object or matched ground-truth point cloud.Euclidean-distance evaluation can yield navigation success despite imperfect mask overlap and slight positional offsets.

C. Real World Language-Grounded Navigation

Real-world experiments use Spot to query and navigate a two-story office building, evaluating floor, room, and object retrieval alongside physical navigation.

  • Real-world setup: The real-world system uses Boston Dynamics Spot with Azure Kinect RGB-D sensing and 3D LiDAR in a two-story office building.The robot collects RGB-D sequences and odometry while traversing rooms with diverse semantic information.
  • Retrieval results: 100% floor, 55.6% room, and 70.7% object retrieval success are achieved across manually labeled real-world goals.Room failures mainly arise from visual similarity among meeting, seminar, and dining rooms.
  • Navigation results: 56.1% object-navigation success is reported across both floors, while all successfully retrieved room and floor concepts are navigated with language instructions.A whiteboard query failed because the robot stopped on the opposite side of a room-separating wall.
  • Representation overhead: HOV-SG reduces memory footprint by as much as 75% on average compared with VLMaps across eight HM3DSem scenes.The comparison includes VLMaps, ConceptGraphs, and HOV-SG; graph structure makes the latter representations smaller than dense counterparts.

E. Ablation Study

The ablation study examines feature clustering and multimodal CLIP-feature fusion as components of HOV-SG’s open-vocabulary segment mapping pipeline.

  • Feature clustering: DBSCAN clustering selects representative pixel-wise CLIP features for each segment to mitigate outliers from CLIP and noise in SAM outputs.The study evaluates this design on the Replica dataset.
  • Ablation configurations: L-CLIP uses masked images including background, whereas M-CLIP uses masked CLIP embeddings without background.These configurations isolate alternative sources of masked-image semantic features.
  • Feature fusion: HOV-SG fuses global, masked-with-background, and masked-without-background CLIP embeddings using a weighted sum.The fusion design is motivated by incorporating salient sub-image features alongside global context.
  • Conclusion and limitation: The paper concludes that HOV-SG surpasses previous baselines in semantic accuracy, open-vocabulary capability, and compactness, but its construction is time-consuming.The method is therefore unsuitable for real-time mapping according to the conclusion.

S.1. METHOD

The supplementary method material describes few-shot hierarchical query parsing, RGB-based graph localization, and open-vocabulary retrieval evaluation.

  • Partial queries: The parser represents omitted hierarchy levels with empty fields, such as [,bedroom,] for a query to go to bed.Fully specified queries are represented as [floor, room, object].
  • Graph localization: HOV-SG localizes the robot in the graph using RGB images, local odometry, and a particle filter initialized within each floor’s free-space map.Global image and object CLIP features support particle updates during localization.
  • Open-vocabulary evaluation: The AUCtop k curve measures performance across evaluation thresholds k as a percentage of the total HM3DSem category count.The example visualization uses scene 00824 with 1624 categories.

S.2. EXPERIMENTAL EVALUATION

The evaluation defines open-vocabulary similarity and 3D semantic-segmentation metrics for comparing predicted and ground-truth labels. AUCtop_k measures ranking quality across large, variably sized category sets.

  • AUCtop_k represents the area under the top-k accuracy curve, with curves closer to the upper-left indicating higher open-vocabulary similarity.
  • The metric normalizes k over the label-category set, which contains 1624 categories for HM3DSem.
  • AUCtop_k provides a dependable measure for large but variably sized label sets.
  • 3D semantic segmentation is evaluated point-wise by comparing predicted labels with ground-truth labels.
  • The reported segmentation metrics are mIOU, F-mIOU, and mAcc, defined from category-level prediction outcomes.

C. Floor and Region Segmentation

The HM3DSem evaluation measures floor, region, room, and object retrieval performance using scene-wise analyses and hierarchical language queries. HOV-SG outperforms ConceptGraphs on object-room-floor and object-room retrieval, while room ambiguity remains a challenge.

  • Floor and Region Segmentation: Scene-wise results show higher precision and recall on smaller scenes containing fewer ground-truth rooms.
  • Room Classification: Room classification compares view embeddings with GPT-3.5 and GPT-4 baselines using exact-text accuracy and human-evaluated qualitative accuracy.
  • Room Classification: The room benchmark uses 14 manually assigned room types, including living room, dining room, kitchen, bathroom, bedroom, office, hallway, and closet.
  • Object Retrieval from Language Queries: The evaluation counts object-retrieval success when IoU exceeds 0.1 and reports top-5 accuracy across 20 frequent object categories.
  • Object Retrieval from Language Queries: 11.69% improvement over ConceptGraphs is reported for object-room-floor queries, while HOV-SG holds a 2.2% advantage for object-room queries.

F. Graph Representation on HM3DSem

HOV-SG represents HM3DSem environments as hierarchical scene graphs connecting floors, rooms, and objects, with visualizations spanning multiple scenes. Real-world trials demonstrate multi-floor language-grounded traversal, while visually similar room types remain a failure source.

  • Graph Representation: The HM3DSem scene graphs connect colored object nodes to blue room nodes and rooms to yellow floor nodes, omitting the multi-floor root for visualization.
  • Graph Representation: Visualization filters out categories containing wall, floor, ceiling, paneling, banner, or overhang, while retaining other predicted object categories.
  • Graph Representation: The visualization procedure removed most ceilings and walls, which the authors report as confirming the accuracy of top-1 predicted open-vocabulary object labels.
  • Graph Representation: HOV-SG scene graphs can support multiple overlapping object masks representing sub-concepts and an additional object hierarchy layer for object parts.
  • Ambiguous Room Labels: Dining, seminar, and meeting rooms often share chairs and large tables, making their visual differentiation difficult and their definitions subjective.
  • Real-World Navigation: The Boston Dynamics Spot completed multi-floor trials from hierarchical queries specifying floor, room, and object, with real-world success rates around 55%.
  • Real-World Navigation: Qualitative navigation visualizations mark successful trials with green masks and failures with red masks.
Loading 2403.17846v2…