PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception

Binh Long Nguyen, Kien Nguyen, Clinton Fookes, Peyman Moghadam

arXiv:2610.07569v1cs.ROcs.AIcs.CV

TL;DR

Dense semantic maps support high-fidelity open-vocabulary perception, while scene graphs represent objects and relations for structured reasoning, but existing approaches typically emphasize one representation. OpenSplatGraph maintains both through a reliability-aware semantic field and persistent graph, and reports competitive performance on benchmarks and real-world robotic tasks.

  • Problem

    Dense semantic maps typically encode semantics as unstructured fields, while scene graphs often rely on sparse geometry, limiting their ability to provide both dense mapping and structured understanding.

    Source: §1 · pp. 1–2

  • Method

    OpenSplatGraph jointly maintains an online Gaussian-based semantic map and persistent scene graph, using observation statistics for query-conditioned object extraction and incremental graph updates.

    Source: §1 · p. 3, Object Association · p. 8

  • Results

    OpenSplatGraph demonstrates competitive performance for online open-vocabulary perception and downstream robotic tasks in standard benchmarks and real-world experiments.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The framework combines dense mapping with persistent object-centric knowledge for language grounding and structured relational reasoning.

    Source: §1 · p. 3, §5 · p. 14

    Object extraction is query-conditioned rather than a complete query-independent inventory, association may be less reliable in clutter or severe viewpoint changes, and VLM-based open-set relations add computational overhead and model dependence.

    Source: §4.6 · p. 14

Abstract

from arXiv · show

Dense 3D mapping with semantic understanding is essential for robotic perception in complex environments. Recent 3D Gaussian Splatting-based mapping approaches enable high-fidelity geometry and efficient open-vocabulary perception, but typically represent semantics as unstructured feature fields that limit object-centric reasoning. In contrast, 3D scene graphs explicitly model objects and their relationships for structured reasoning, but are commonly constructed from sparse geometric representations that do not fully exploit dense semantic maps. In this work, we present OpenSplatGraph, a unified framework that constructs persistent 3D scene graphs directly from an online Gaussian-based open-vocabulary semantic map. The proposed framework augments the dense semantic map with a reliability-aware semantic field that maintains lightweight observation statistics for confidence-aware, query-conditioned object extraction. Extracted object instances are associated with persistent graph nodes, allowing object attributes and relationships to be incrementally updated across observations and queries. By tightly coupling dense semantic mapping with persistent object-centric representations, our framework supports both language-guided object grounding and structured relational reasoning while preserving the geometric fidelity of Gaussian-based mapping. Comprehensive evaluations on standard 3D scene understanding benchmarks and real-world robotic experiments demonstrate that OpenSplatGraph achieves competitive performance for online open-vocabulary perception and downstream robotic tasks. Project page: https://csiro-robotics.github.io/OpenSplatGraph.

1 Introduction

Dense semantic maps and scene graphs offer complementary capabilities, but existing approaches typically emphasize one representation over the other. OpenSplatGraph maintains both, grounding a persistent scene graph in an online open-vocabulary Gaussian semantic map.

1 Introduction

  • OpenSplatGraph jointly maintains dense semantic mapping and a persistent scene graph to combine high-fidelity mapping with structured scene understanding.The dense map supports language-grounded perception, while the graph accumulates persistent object-centric knowledge for reasoning.Source: Figure 1 · p. 2, §1 · p. 3
  • A reliability-aware semantic field uses lightweight per-voxel observation statistics for robust query-conditioned object extraction under partial observations and semantic ambiguity.Source: §1 · p. 3
  • Extracted object instances receive persistent identities and are incrementally incorporated into the scene graph, accumulating object attributes and relationships across queries.Source: §1 · p. 3
  • The framework demonstrates competitive performance for online open-vocabulary perception and downstream robotic tasks in standard benchmarks and real-world experiments.Source: §1 · p. 3

2 Related Work

Related work develops open-vocabulary perception, Gaussian-based mapping, and scene graphs, while highlighting a gap between dense semantic maps and structured object-centric representations.

2.1 Open-Vocabulary 3D Scene Understanding

  • Open-vocabulary methods align visual representations with language and extend querying to Gaussian primitives, but their unstructured semantic fields limit object-centric and relational reasoning.Examples include LangSplat, LEGaussian, OpenGaussian, and InstanceGaussian.Source: §2.1 · pp. 3–4

2.2 3D Gaussian-Based Scene Mapping

  • Gaussian-based SLAM methods support online dense mapping and high-fidelity rendering, while semantic fusion adds geometrically accurate, semantically rich maps without persistent object-centric representations.Examples include SplaTAM, MonoGS, GS-ICP SLAM, and OpenGS-Fusion.Source: §2.2 · p. 4

2.3 3D Scene Graphs

  • Scene graphs offer interpretable object-and-relationship representations, but many open-vocabulary methods rely on point clouds or meshes and weakly ground relations in dense 3D geometry.OpenSplatGraph maintains both the dense semantic field and scene graph, using the former for object extraction and the latter for relational reasoning.Source: §2.3 · p. 4

3 Methodology

OpenSplatGraph jointly maintains an online dense open-vocabulary semantic field and a persistent 3D scene graph. It extracts objects on demand from language queries and incrementally updates the graph to support structured reasoning.

3 Methodology

  • The framework keeps dense semantic mapping and a persistent scene graph together, using the map for query-conditioned object extraction and the graph for accumulated object-centric reasoning.Source: §3 · pp. 4–5, Figure 2 · p. 5

3.1 Reliability-Aware Semantic Field

  • Each voxel in the hybrid map is associated with Gaussian primitives and stores fused semantic features, observation count, accumulated confidence, and global coordinates.Source: Semantic Representation · p. 6
  • Per-voxel observation count, accumulated confidence, and semantic disagreement characterize reliability, while instance confidence favors well-observed, semantically consistent regions and ranks or filters candidates without supervised calibration.Source: Reliability-Aware Semantic Modeling · p. 6
  • Language-conditioned extraction thresholds voxel-query similarity, clusters the selected coordinates with DBSCAN, and ranks instances using semantic similarity and reliability-based confidence.Cluster size breaks ties among candidates.Source: Reliability-Aware Semantic Modeling · pp. 6–7, Query-conditioned Object Extraction · p. 7

3.2 Persistent 3D Scene Graph Construction

  • New instances reuse the nearest unmatched prior centroid with the same label within a distance threshold, or receive a new ID; graph nodes retain IDs as attributes are updated.Source: Object Association · p. 8, Nodes · p. 8, §3.2 · p. 8
  • Graph edges combine online geometry-based spatial relations with optional VLM-inferred open-set semantic relations aggregated by multi-view voting.Spatial relations include near, on/under, inside, and relative ordering.Source: Edges · p. 9

4 Experiments

The experiments assess OpenSplatGraph’s semantic field, persistent scene graph, complete robotic system, and component contributions. Results cover semantic segmentation and object grounding benchmarks, real-world robotic tasks, and stated framework limitations.

4.2 3D Semantic Segmentation

  • OpenSplatGraph consistently surpasses ConceptFusion and OpenGS-Fusion on online ScanNet and Replica segmentation, and is competitive with InstanceGaussian among offline methods.The comparison reports mIoU and mAcc; Table 1 covers both datasets.Source: Table 1 · p. 10, §4.2 · pp. 10–11

4.1 Implementation Details

  • The proposed reliability-aware semantic field is evaluated for open-vocabulary 3D semantic segmentation on ScanNet and Replica using mIoU and mAcc, against 3D-GS and point-cloud-based methods.Replica’s 88 semantic categories are remapped to the ScanNet20 and NYUv2-40 label sets for consistent evaluation.Source: §4.2 · p. 10

4.2 3D Semantic Segmentation

  • Qualitative segmentation results show more spatially coherent instances aligned to boundaries, particularly for thin structures and partially observed objects.Examples include shelves, doors, and blinds.Source: §4.2 · p. 11, Figure 3 · p. 11

4.3 3D Object Grounding

  • OpenSplatGraph improves Recall@k over ConceptGraphs in several Replica grounding settings, with the largest gains for affordance and negation queries.The evaluation compares CLIP- and LLM-based retrieval across descriptive, affordance, and negation queries.Source: Settings · p. 11, Results · p. 12, Table 2 · p. 12

4.4 Real-world Robotic Experiments

  • On a mobile manipulator, the system supports language-guided navigation and pick-and-place by grounding queries in the persistent graph and localizing objects in the dense map.The experiments use a Stretch 3 robot in a real-world indoor environment.Source: §4.4 · pp. 12–13, Figure 5 · p. 13

4.5 Ablation Study

  • The progressive ablation attributes segmentation improvement to reliability-aware modeling, grounding improvement to persistent identities, and the largest relational-grounding gain to graph reasoning.The full framework performs best across all reported metrics.Source: §4.5 · pp. 13–14, Table 3 · p. 14

4.6 Limitations

  • Object extraction is query-conditioned rather than inventory-based, association may weaken in clutter or severe viewpoint changes, and external VLM relation inference adds compute and model dependence.The authors identify query-independent proposals and more robust association as future directions.Source: §4.6 · p. 14

5 Conclusion

OpenSplatGraph jointly maintains a reliability-aware semantic field and persistent scene graph for online robotic perception. Benchmark and real-world evaluations report competitive perception performance and support for language-guided navigation and manipulation.

5 Conclusion

  • OpenSplatGraph jointly maintains a reliability-aware open-vocabulary semantic field and persistent 3D scene graph for online robotic perception.The field supports query-conditioned object extraction, while the graph accumulates object-centric knowledge for relational reasoning.Source: §5 · p. 14
  • Experiments report competitive benchmark performance in semantic segmentation and object grounding, and validate language-guided navigation and manipulation in real-world robotics.Source: §5 · p. 14
  • Future work will address the discussed limitations and explore multi-step task planning in more complex environments.Source: §5 · p. 14

A Additional 3D Semantic Segmentation Results

Additional segmentation comparisons report higher mIoU for OpenSplatGraph than GaussianGraph on both datasets; DynamicGSG’s differently mapped Replica results are reference-only.

A Additional 3D Semantic Segmentation Results

B Direct Scene-Graph Evaluation

Direct evaluation against ConceptGraphs assesses graph correctness and identity consistency. OpenSplatGraph reports higher node and edge precision and a lower duplicate-node rate.

B Direct Scene-Graph Evaluation

Related papers

Loading 2610.07569v1…