Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

• Large-scale open-vocabulary scene representations must provide object-centric hierarchies, storage efficiency, actionability, and true open-vocabulary querying.Source: §I · p. 1
• HOV-SG indexes floors, rooms, and objects with open-vocabulary features in hierarchical 3D scene graphs spanning multi-story environments.Source: §I · p. 1
• HOV-SG decomposes abstract queries across hierarchy levels and supports object retrieval and long-horizon navigation in large-scale indoor environments.Source: §I · p. 2
• The work evaluates semantic segmentation on Replica and ScanNet, analyzes scene-graph properties on Habitat-Matterport, and conducts real-world multi-floor navigation experiments.Source: §I · p. 2
• Existing semantic maps often use fixed category sets, while open-vocabulary feature maps can incur substantial storage overhead by assigning embeddings to geometric elements.Source: A. Semantic 3D Mapping · p. 2

Scene Graphs for Planning

• HOV-SG targets indexing of floor-, room-, and object-level concepts from natural-language queries while supporting semantic and spatial robot navigation.Source: §III · p. 3

3D Segment-Level Open-Vocabulary Mapping

• The segment-level map assigns each 3D segment an open-vocabulary feature rather than storing an independent visual-language feature for every 3D point.Source: A. 3D Segment-Level Open-Vocabulary… · p. 3
• Segments are merged through a fully connected overlap graph, where highly connected subgraphs are subsequently combined.Source: A. 3D Segment-Level Open-Vocabulary… · p. 3

3D Scene Graph Construction

• Floor boundaries come from filtered peaks in a height-density histogram, while room masks are extracted with the Watershed algorithm.Source: Figure 3 · p. 4

3D Segment-Level Open-Vocabulary Mapping

• The feature pipeline fuses CLIP embeddings from the full frame, a mask crop, and a background-excluded mask crop using a weighted sum.Source: A. 3D Segment-Level Open-Vocabulary… · p. 4
• The fused mask features are projected into global 3D coordinates, associated with nearby reference points, averaged per point, and used to infer features for merged 3D segments.Source: A. 3D Segment-Level Open-Vocabulary… · p. 4

Hierarchical graph

• HOV-SG formalizes nodes for the scene root, floors, rooms, and objects, with hierarchical edges connecting these concept levels.Each non-root node stores its concept point cloud and associated open-vocabulary features.Source: B. 3D Scene Graph Construction · p. 4

Room features

• Room nodes receive representative CLIP view embeddings from RGB-D observations whose camera poses lie inside the room segment.K-means distills the observations into k representative embeddings, which support room-category voting through cosine similarity.Source: B. 3D Scene Graph Construction · pp. 4–5, Figure 4 · p. 5

Object association

• Object segments are assigned to rooms using bird’s-eye-view point-cloud overlap, with nearest-room assignment when no overlap exists.This associates object-level geometry with the room hierarchy.Source: B. 3D Scene Graph Construction · p. 5

Navigational graphs

• Single-floor navigation constructs a navigable area by removing obstacles from camera poses and floor point-cloud projections, then builds a Voronoi graph.The resulting graph represents traversable structure within each floor.Source: Figure 5 · p. 5
• Cross-floor navigation extracts camera positions in stair regions and connects this subgraph to the corresponding floor-level Voronoi graphs.This extends actionable navigation across multiple floors.Source: Figure 5 · p. 5

Query decomposition

• HOV-SG decomposes complex instructions such as “find the toilet in the bathroom on floor 2” into floor-, room-, and object-level queries.This extends navigation goals beyond isolated objects to spatial concepts such as rooms and floors.Source: C. Navigation with Scene Graph · p. 5

Hierarchical grounding

• The system sequentially scores query concepts against floors, rooms, and objects to progressively narrow the solution corridor.Cosine similarity identifies a target node at each hierarchy level.Source: C. Navigation with Scene Graph · p. 6

Path planning

• After target identification, HOV-SG uses its navigational graph to plan a path from the starting pose to the target destination.The graph supports actionable grounding of decomposed language queries.Source: C. Navigation with Scene Graph · p. 6

Evaluation

• The experiments examine semantic segmentation, scene-graph accuracy, large-scale real-world language-grounded navigation, and compact memory usage.These goals span ScanNet, Replica, and Habitat Matterport 3D Semantics evaluations.Source: §IV · p. 6

Benchmark evaluation

• HOV-SG is compared with ConceptFusion and ConceptGraphs on open-vocabulary 3D semantic segmentation using ScanNet and Replica scenes.The evaluation uses different CLIP backbones and reports results in Table I.Source: A. 3D Semantic Segmentation on ScanNet… · p. 6
• HOV-SG outperforms the open-vocabulary baselines by a large margin on both mIOU and F-mIOU.Predicted object labels are projected to ground-truth points through five-nearest-neighbor majority voting for evaluation.Source: A. 3D Semantic Segmentation on ScanNet… · p. 6

Qualitative results

• On Replica qualitative examples, only HOV-SG predicts the door with correct boundaries at position A and the sofa correctly at position B.The figure legend displays ten of the dataset’s 101 categories.Source: Figure 6 · p. 7

Scene-graph structure

• 100% accuracy is achieved for retrieving the number of floors in both single-floor and multi-floor HM3DSem scenes.Region segmentation recall reaches 83.59%, compared with 77.55% for Hydra, while precision is slightly lower.Source: 1) Floor and Class-Agnostic Room… · p. 7

Room semantics

• The room-classification evaluation compares HOV-SG’s view-embedding method with GPT-3.5 and GPT-4 baselines under privileged and unprivileged settings.Exact text equality and human-assessed semantic correctness are evaluated separately.Source: 2) Semantic Room Classification · pp. 7–8

Metric

• AUCtop-k measures the area under the top-k accuracy curve between predicted and ground-truth categories across the 1624-category label set.It reflects how many ranked candidate labels are needed on average before the ground-truth category appears.Source: 3) Object-Level Semantics · pp. 8–9

Evaluation protocol

• The evaluation uses linear assignment between predicted and ground-truth objects and retains predicted objects with IoU greater than 50%.Because VLMaps does not predict masks, it uses HOV-SG’s predicted object masks for comparison.Source: 3) Object-Level Semantics · p. 9

Results

• HOV-SG achieves 84.88% AUCtop-k, compared with 84.07% for ConceptGraphs.HOV-SG outperforms ConceptGraphs through the top-100 highest-ranking classes.Source: 3) Object-Level Semantics · p. 9
• VLMaps identifies only 5% of objects correctly within its top-5 choices and reaches the correct class within top-500 predictions in 40.02% of cases.The passage attributes this weaker result presumably to dense feature aggregation and dependence on the fine-tuned LSeg model.Source: 3) Object-Level Semantics · p. 9

Query decomposition and retrieval

• Hierarchical queries are decomposed into concepts such as [floor 2, living room, pillow] or [-, kitchen, bottle] before CLIP-based scoring.The system queries the most suitable floor, room, and object in sequence.Source: 4) Hierarchical Concept Retrieval · p. 9

Retrieval results

• 11.69% higher performance is reported for object-room-floor queries, while object-room queries gain 2.2% over the augmented ConceptGraphs baseline.ConceptGraphs struggles on larger scenes and more detailed queries.Source: 4) Hierarchical Concept Retrieval · p. 9

Evaluation protocol

• Top-5 retrieval accuracy is evaluated over 20 frequent object categories, with success requiring IoU > 0.1.The evaluated categories include objects such as pillows, doors, lamps, chairs, tables, towels, and plants.Source: 4) Hierarchical Concept Retrieval · p. 10

Navigation evaluation

• Navigation success is counted when the robot reaches within approximately 1 m of the sought object or matched ground-truth point cloud.Euclidean-distance evaluation can yield navigation success despite imperfect mask overlap and slight positional offsets.Source: 4) Hierarchical Concept Retrieval · p. 10

Real-world setup

• The real-world system uses Boston Dynamics Spot with Azure Kinect RGB-D sensing and 3D LiDAR in a two-story office building.The robot collects RGB-D sequences and odometry while traversing rooms with diverse semantic information.Source: C. Real World Language-Grounded… · p. 10, Figure 9 · p. 10

Retrieval results

• 100% floor, 55.6% room, and 70.7% object retrieval success are achieved across manually labeled real-world goals.Room failures mainly arise from visual similarity among meeting, seminar, and dining rooms.Source: C. Real World Language-Grounded… · p. 10

Navigation results

• 56.1% object-navigation success is reported across both floors, while all successfully retrieved room and floor concepts are navigated with language instructions.A whiteboard query failed because the robot stopped on the opposite side of a room-separating wall.Source: C. Real World Language-Grounded… · p. 10

Representation overhead

• HOV-SG reduces memory footprint by as much as 75% on average compared with VLMaps across eight HM3DSem scenes.The comparison includes VLMaps, ConceptGraphs, and HOV-SG; graph structure makes the latter representations smaller than dense counterparts.Source: D. Representation Storage Overhead… · p. 11, C. Real World Language-Grounded… · p. 11

Feature clustering

• DBSCAN clustering selects representative pixel-wise CLIP features for each segment to mitigate outliers from CLIP and noise in SAM outputs.The study evaluates this design on the Replica dataset.Source: E. Ablation Study · p. 11

Ablation configurations

• L-CLIP uses masked images including background, whereas M-CLIP uses masked CLIP embeddings without background.These configurations isolate alternative sources of masked-image semantic features.Source: E. Ablation Study · p. 11

Feature fusion

• HOV-SG fuses global, masked-with-background, and masked-without-background CLIP embeddings using a weighted sum.The fusion design is motivated by incorporating salient sub-image features alongside global context.Source: E. Ablation Study · p. 11

Conclusion and limitation

• The paper concludes that HOV-SG surpasses previous baselines in semantic accuracy, open-vocabulary capability, and compactness, but its construction is time-consuming.The method is therefore unsuitable for real-time mapping according to the conclusion.Source: §V · p. 11

Partial queries

• The parser represents omitted hierarchy levels with empty fields, such as [,bedroom,] for a query to go to bed.Fully specified queries are represented as [floor, room, object].Source: A. Robot Navigation Prompts · p. 14

Graph localization

• HOV-SG localizes the robot in the graph using RGB images, local odometry, and a particle filter initialized within each floor’s free-space map.Global image and object CLIP features support particle updates during localization.Source: B. Semantic Localization · p. 14

Open-vocabulary evaluation

• The AUCtop k curve measures performance across evaluation thresholds k as a percentage of the total HM3DSem category count.The example visualization uses scene 00824 with 1624 categories.Source: Figure S.2 · p. 15

• AUCtop_k represents the area under the top-k accuracy curve, with curves closer to the upper-left indicating higher open-vocabulary similarity.Source: §S.2 · p. 15
• The metric normalizes k over the label-category set, which contains 1624 categories for HM3DSem.Source: §S.2 · p. 15
• AUCtop_k provides a dependable measure for large but variably sized label sets.Source: §S.2 · p. 15
• 3D semantic segmentation is evaluated point-wise by comparing predicted labels with ground-truth labels.Source: B. Evaluation Metrics for 3D Semantic… · p. 15
• The reported segmentation metrics are mIOU, F-mIOU, and mAcc, defined from category-level prediction outcomes.Source: B. Evaluation Metrics for 3D Semantic… · p. 15

Floor and Region Segmentation

• Scene-wise results show higher precision and recall on smaller scenes containing fewer ground-truth rooms.Source: C. Floor and Region Segmentation · p. 15

Room Classification

• Room classification compares view embeddings with GPT-3.5 and GPT-4 baselines using exact-text accuracy and human-evaluated qualitative accuracy.Source: D. Room Classification · p. 16, Table S.2 · p. 16
• The room benchmark uses 14 manually assigned room types, including living room, dining room, kitchen, bathroom, bedroom, office, hallway, and closet.Source: D. Room Classification · p. 16

Object Retrieval from Language Queries

• The evaluation counts object-retrieval success when IoU exceeds 0.1 and reports top-5 accuracy across 20 frequent object categories.Source: E. Language-Grounded Navigation with… · p. 16
• 11.69% improvement over ConceptGraphs is reported for object-room-floor queries, while HOV-SG holds a 2.2% advantage for object-room queries.Source: E. Language-Grounded Navigation with… · pp. 16–17, Table S.3 · p. 16

Graph Representation

• The HM3DSem scene graphs connect colored object nodes to blue room nodes and rooms to yellow floor nodes, omitting the multi-floor root for visualization.Source: F. Graph Representation on HM3DSem · p. 17, Figure S.4 · p. 18
• Visualization filters out categories containing wall, floor, ceiling, paneling, banner, or overhang, while retaining other predicted object categories.Source: F. Graph Representation on HM3DSem · p. 17
• The visualization procedure removed most ceilings and walls, which the authors report as confirming the accuracy of top-1 predicted open-vocabulary object labels.Source: F. Graph Representation on HM3DSem · p. 17
• HOV-SG scene graphs can support multiple overlapping object masks representing sub-concepts and an additional object hierarchy layer for object parts.Source: F. Graph Representation on HM3DSem · p. 17

Ambiguous Room Labels

• Dining, seminar, and meeting rooms often share chairs and large tables, making their visual differentiation difficult and their definitions subjective.Source: G. Real-World Ambiguous Room Labels · p. 17, Figure S.3 · p. 17

Real-World Navigation

• The Boston Dynamics Spot completed multi-floor trials from hierarchical queries specifying floor, room, and object, with real-world success rates around 55%.Source: H. Real-World Qualitative Navigation… · pp. 17, 19, Figure S.5 · p. 19
• Qualitative navigation visualizations mark successful trials with green masks and failures with red masks.Source: Figure S.6 · p. 20