PaperlayerView exact revision on arXiv

Source-linked AI summary

OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding

Siting Zhu, Ziyun Lu, Guangming Wang, Chenguang Huang, Yongbo Chen, I-Ming Chen, Wolfram Burgard, Hesheng Wang

arXiv:2603.16301v2cs.RO

TL;DR

Robotic scene understanding must handle novel objects and incremental exploration, but existing graph methods often require complete prebuilt semantic maps. OGScene3D addresses this with confidence-aware Gaussian mapping, hierarchical and long-term refinement, and progressive graph construction, reporting stronger 2D and 3D segmentation results than baselines.

  • Problem

    Existing scene-understanding methods are limited by closed vocabularies, local semantic updates, and offline graph construction requiring prebuilt maps, despite robots encountering novel objects.

    Source: §I · p. 1

  • Method

    OGScene3D uses confidence-based Gaussian semantics, hierarchical local-to-global optimization, long-term historical refinement, and progressive graph construction for incremental mapping.

    Source: §I · p. 2, 2) Local 3D Optimization · p. 6

  • Results

    OGScene3D improves novel-view segmentation by 16% average mIoU and 22% average mAcc, with up to 36% and 28% average-precision gains on Replica and ScanNet 3D segmentation.

    Source: 1) 2D Semantic Segmentation · p. 12, 2) 3D Semantic Segmentation · p. 13

  • Takeaways & Limitations

    The system supports continuously updated open-vocabulary semantic maps and scene graphs during progressive environmental exploration.

    Source: §I · p. 2, A. Conclusion · p. 16

    OGScene3D models spatial relationships but does not cover complex inter-entity relationships such as functional associations.

    Source: B. Limitation and Future Work · p. 16

Abstract

from arXiv · show

Open-vocabulary scene understanding is crucial for robotic applications, enabling robots to comprehend complex 3D environmental contexts and supporting various downstream tasks such as navigation and manipulation. However, existing methods require pre-built complete 3D semantic maps to construct scene graphs for scene understanding, which limits their applicability in robotic scenarios where environments are explored incrementally. To address this challenge, we propose OGScene3D, an open-vocabulary scene understanding system that achieves accurate 3D semantic mapping and scene graph construction incrementally. Our system employs a confidence-based Gaussian semantic representation that jointly models semantic predictions and their reliability, enabling robust scene modeling. Building on this representation, we introduce a hierarchical 3D semantic optimization strategy that achieves semantic consistency through local correspondence establishment and global refinement, thereby constructing globally consistent semantic maps. Moreover, we design a long-term global optimization method that leverages temporal memory of historical observations to enhance semantic predictions. By integrating 2D-3D semantic consistency with Gaussian rendering contribution, this method continuously refines the semantic understanding of the entire scene. Furthermore, we develop a progressive graph construction approach that dynamically creates and updates both nodes and semantic relationships, allowing continuous updating of the 3D scene graphs. Extensive experiments on widely used datasets and real-world scenes demonstrate the effectiveness of our OGScene3D on open-vocabulary scene understanding.

I. INTRODUCTION

OGScene3D targets open-vocabulary 3D understanding for incrementally explored robotic environments, where existing methods rely on closed vocabularies or prebuilt maps. It combines confidence-aware Gaussian semantics, hierarchical and long-term optimization, and progressive graph construction.

I. INTRODUCTION

  • Open-vocabulary scene understanding matters for robotics because robots encounter novel objects during manipulation and navigation, while many prior methods assume predefined categories.Source: §I · p. 1
  • Existing approaches trade semantic expressiveness, precision, and memory efficiency, while frame-wise updates leave global inconsistencies and offline graph construction requires prebuilt semantic maps.Source: §I · pp. 1–2
  • OGScene3D uses a confidence-based Gaussian semantic representation that models labels and reliability, filtering uncertain predictions caused by segmentation ambiguities and geometric errors.Source: §I · p. 2
  • Hierarchical optimization combines local frame-to-scene associations with global refinement, while long-term optimization uses historical information to improve semantic consistency and sparsely observed regions.Source: §I · p. 2
  • Progressive graph construction dynamically updates semantic nodes and relationships as exploration proceeds, enabling incrementally built scene graphs instead of offline graph generation.Source: §I · p. 2

A. 3D Representation and Semantic Mapping

  • The approach builds on explicit 3D representations including points, surfels, voxels, TSDFs, meshes, NeRFs, and 3DGS, while addressing the closed-set limitations of prior semantic mapping systems.Source: A. 3D Representation and Semantic… · pp. 2–3

B. 3D Scene Graphs

OGScene3D extends open-vocabulary scene graphs to incremental exploration by combining progressive graph updates with confidence-aware Gaussian semantic mapping. Its representation and optimization pipeline supports semantic rendering, online mapping, and continuously refined spatial relationships.

B. 3D Scene Graphs

  • Existing open-vocabulary scene graphs commonly construct relationships only after processing the full sequence and therefore depend on prebuilt semantic maps.Source: B. 3D Scene Graphs · p. 3
  • Progressive graph construction adds nodes, establishes relationships, and applies confidence-based node updates as the scene is explored.Source: B. 3D Scene Graphs · p. 3, §III · p. 3

III. METHOD

A. Confidence-based Scene Representation

B. Hierarchical 3D Semantic Optimization

OGScene3D combines open-set 2D segmentation with hierarchical 3D optimization to build semantically consistent Gaussian maps incrementally. Local association uses spatial-semantic evidence and confidence updates to handle ambiguous, novel, and repeatedly observed regions.

B. Hierarchical 3D Semantic Optimization

  • Hierarchical 3D semantic optimization combines local cross-frame association with global refinement to maintain consistency across the scene.The method first establishes correspondences between views, then refines semantics globally.Source: B. Hierarchical 3D Semantic Optimization · p. 5

1) Open-set Segmentation:

  • Open-set segmentation fuses SAM proposals with YOLO-World detections, retaining detector segments and adding nonoverlapping SAM regions for broader coverage.Detection prompts improve object-level precision, while supplementary SAM segments cover missed objects.Source: 1) Open-set Segmentation · p. 5

2) Local 3D Optimization:

  • New Gaussian segments are matched to existing Gaussians with KNN, and ambiguity ratio r(m) plus coherence p(m) determine whether labels are rejected, associated, or treated as novel.Ambiguous segments are excluded; coherent matches inherit existing labels, while low-coherence segments receive novel labels.Source: Figure 3 · p. 5, 2) Local 3D Optimization · pp. 5–6
  • Each Gaussian stores a semantic label and confidence, which is initialized from SAM confidence and updated using spatial correspondence and cross-view semantic agreement.Confidence rises for matching observations and falls for differing labels; unlabeled or novel segments are excluded from these updates.Source: 2) Local 3D Optimization · p. 6

3) Global 3D Refinement:

Global refinement identifies unreliable semantic clusters and individual Gaussians, then propagates labels from nearby high-confidence regions while preserving geometric detail and uncertainty information.

3) Global 3D Refinement:

  • Cluster-level refinement replaces low-average-confidence cluster labels with the most frequent label among nearby high-confidence Gaussians.This combines confidence filtering with spatial proximity while preserving the cluster geometry.Source: 3) Global 3D Refinement · p. 6
  • Individual refinement corrects remaining low-confidence Gaussians by selecting the nearest high-confidence candidate within a search radius.The procedure targets uncertain regions such as object boundaries after cluster-level processing.Source: 3) Global 3D Refinement · p. 7
  • Refined Gaussians retain local geometric details, while their confidence may be set to 0.5 to prevent immediate overwriting during later optimization.The refinement transfers reliable semantic information while retaining uncertainty from earlier stages.Source: 3) Global 3D Refinement · p. 7

4) Loss Functions:

The optimization jointly uses rendered color and depth agreement with RGB-D observations, with SSIM included in the color loss and weighted terms forming the overall objective.

4) Loss Functions:

  • Depth and color losses compare rendered outputs with input RGB-D values to optimize Gaussian geometry and appearance.The rendered color and depth are evaluated for the camera pose in world coordinates.Source: 4) Loss Functions · p. 7
  • SSIM is added to the color loss to strengthen image-appearance supervision.Source: 4) Loss Functions · p. 7
  • The overall objective is the weighted sum of the component losses.Source: 4) Loss Functions · p. 7

C. Long-term Global Optimization

Long-term global optimization uses temporally distributed observations to improve semantic confidence in underobserved regions. It combines 2D-3D consistency with rendering contribution while updating confidence rather than established labels.

C. Long-term Global Optimization

  • A memory buffer stores observations collected at different times and covering complementary spatial regions to address uncertainty from sparse viewpoints.Source: C. Long-term Global Optimization · p. 7
  • A long-term global optimization updates semantic confidence using 2D-3D semantic consistency and Gaussian rendering contribution, improving semantic understanding of the entire scene in underobserved areas.It updates confidence rather than semantic labels to maintain scene-representation stability and identify unreliable regions for subsequent global refinement.Source: C. Long-term Global Optimization · p. 7
  • The long-term stage updates confidence but not semantic labels, preserving established representations while identifying unreliable regions for later global refinement.Source: C. Long-term Global Optimization · p. 7

1) Memory Buffer Construction:

The system builds a compact memory buffer from geometrically distinctive observations, then uses historical views, 2D–3D consistency, and rendering contribution to refine Gaussian semantic confidence.

1) Memory Buffer Construction:

  • Keyframes enter the memory buffer only when viewpoint disparity and geometric overlap criteria both indicate substantial new scene coverage.The resulting buffer is compact while remaining geometrically representative for long-term optimization.Source: 1) Memory Buffer Construction · p. 8

3) Gaussian Rendering Contribution:

2) 2D-3D Semantic Consistency:

  • 2D–3D semantic consistency is measured by projecting the global map into memory-buffer views, computing class-wise IoU, and transferring pixel confidence to Gaussians through KNN matching.This produces Gaussian-wise confidence factors from alignment between SAM segmentation and the rendered 3D semantic map.Source: 2) 2D-3D Semantic Consistency · p. 8

3) Gaussian Rendering Contribution:

  • Gaussian rendering contribution is estimated with gradients of squared and linear losses, so Gaussians influencing rendered pixels receive adaptive confidence weights.The squared loss emphasizes high-rendered-value pixels, whereas the linear loss provides an overall-confidence baseline.Source: 3) Gaussian Rendering Contribution · p. 8

4) Long-term Confidence Update:

Historical observations increase confidence for consistent Gaussian labels and decrease it for inconsistent ones, while progressive graph construction initializes semantic entities and their spatial relationships from early keyframes.

4) Long-term Confidence Update:

  • Consistent historical labels receive positive confidence updates, whereas inconsistent labels receive negative updates using both 2D–3D consistency and rendering contribution.Temporally distant observations improve underobserved regions and produce a more complete, consistent semantic representation.Source: 4) Long-term Confidence Update · p. 9

D. Progressive Graph Construction

  • The scene graph represents semantic regions as captioned nodes and inter-entity relationships as edges, enabling scene understanding beyond SAM region identifiers.Graph construction proceeds through initialization followed by updates.Source: D. Progressive Graph Construction · p. 9

1) Graph Initialization:

  • Initialization uses the first 12 keyframes to generate multi-view entity captions, initialize node confidence, and establish spatially supported edges.Entity observations combine the largest contextual crop with three large masked views when available.Source: 1) Graph Initialization · pp. 9–10
  • Spatial relationships are first proposed from point-cloud proximity and interaction ratios, then semantically labeled by GPT-4o using edge prompts.The design keeps entity categories open-vocabulary while favoring standardized spatial relations for efficient robotic query matching.Source: 1) Graph Initialization · p. 10

2) Graph Update:

Graph updates selectively refine uncertain or newly discovered nodes and recompute only relationships involving those nodes, progressively improving scene understanding during exploration.

2) Graph Update:

  • Nodes are updated when their confidence falls below a threshold or when new entities are discovered, rather than at fixed intervals for every node.Selected nodes receive refreshed multi-view representations using observations with larger projected areas.Source: 2) Graph Update · p. 10
  • Updated multi-view inputs generate refined captions, balancing computational efficiency with semantic accuracy.Geometric contribution prioritizes views with better visibility and detail capture.Source: 2) Graph Update · p. 10
  • Only edges connected to updated or newly added nodes are reevaluated through spatial adjacency analysis, avoiding full-graph recomputation.This targeted strategy supports progressive enhancement as exploration reveals new observations.Source: 2) Graph Update · p. 10

IV. EXPERIMENTS

Experiments evaluate semantic mapping on Replica and ScanNet, graph construction on 3RScan, and practical behavior in real-world scenes using segmentation and graph-recall metrics against open-vocabulary baselines.

1) Datasets:

  • The evaluation spans eight Replica scenes, five ScanNet scenes, four 3RScan scenes, and additional real-world scenes.Replica and ScanNet provide semantic segmentation ground truth, while 3RScan provides relationship ground truth.Source: 1) Datasets · p. 11

2) Evaluation Metrics:

  • Semantic segmentation is measured with mIoU, F-mIoU, and mAcc, while scene graph construction is evaluated with Recall.These metrics cover both semantic mapping quality and relational graph accuracy.Source: 2) Evaluation Metrics · p. 11

3) Baselines:

  • Baselines include traditional representation-based and 3DGS-based mapping methods, with ConceptGraphs used primarily for scene graph construction comparison.Novel-view segmentation comparisons focus on 3DGS methods because traditional approaches produce sparse 2D results.Source: 3) Baselines · p. 11

4) Implementation Details:

  • The system uses fixed semantic thresholds, performs global refinement and long-term optimization every six keyframes, and updates the graph every 12 keyframes.Node updates are triggered below confidence threshold τupdate = τlow = 0.3.Source: 4) Implementation Details · pp. 11–12

B. Semantic Mapping Evaluation:

  • In Replica qualitative comparisons, baselines perform poorly on large background regions and fine-grained details.The cited passage describes the comparison context but does not state a numeric result.Source: Figure 6 · p. 12

1) 2D Semantic Segmentation:

OGScene3D improves open-vocabulary 2D and 3D semantic mapping, scene-graph relationship estimation, and incremental real-world scene understanding.

1) 2D Semantic Segmentation:

  • 16% higher mIoU and 22% higher mAcc than baselines across test scenes in novel-view 2D semantic segmentation.The method better preserves background regions, fine-grained structures, and semantic boundaries in Replica and ScanNet comparisons.Source: 1) 2D Semantic Segmentation · pp. 12–13

2) 3D Semantic Segmentation:

C. Scene Graph Evaluation:

  • Confidence-based updating enables superior open-set relationship estimation by refining scene-graph edges as additional viewpoints become available.The comparison extends ConceptGraphs to the full relationship vocabulary present in 3RScan for fairness.Source: C. Scene Graph Evaluation · p. 14, Table IV · p. 14

D. Real World Experiment

E. Runtime Analysis

OGScene3D balances mapping efficiency and accuracy while its ablations show that global refinement and historical observations improve semantic consistency during incremental exploration.

E. Runtime Analysis

  • The method requires significantly less computational time than baselines while simultaneously achieving higher mapping accuracy.Runtime excludes the shared SAM preprocessing cost and measures complete-scene processing duration.Source: E. Runtime Analysis · p. 15, Table V · p. 15

1) Effectiveness of Hierarchical 3D Semantic Optimization:

2) Effectiveness of Long-term Global Optimization:

A. Conclusion

  • OGScene3D incrementally constructs semantic maps and scene graphs through confidence-based Gaussian semantics, hierarchical refinement, historical optimization, and progressive graph updates.These components support continuous semantic understanding during robotic exploration.Source: A. Conclusion · p. 16

B. Limitation and Future Work

  • The system currently models spatial relationships but does not explore complex inter-entity relationships such as functional associations.Future work targets comprehensive multi-relational scene understanding for real-time robotic deployment.Source: B. Limitation and Future Work · p. 16

Related papers

Loading 2603.16301v2…