PaperlayerView exact revision on arXiv

Source-linked AI summary

DSG: Dynamic 3D Scene Graph Construction for Embodied Agents in Changing Indoor Environments

Ming Liao, Chao Ye, Jianing Fei, Weiyang Lin

arXiv:2609.00619v1cs.RO

TL;DR

DSG addresses the problem of scene graphs becoming inconsistent when indoor objects move, combining dynamic object-node updates with visual-context-enhanced spatial reasoning. It introduces Dyn-THOR for evaluation and reports consistent gains in object node construction and relationship reasoning across simulated and real-world scenes.

  • Problem

    Existing scene-graph methods insufficiently maintain consistency after object relocation and underuse observation-specific visual context for ambiguous spatial relationships.

  • Method

    DSG uses dual-view rendering for object change detection, multi-granularity VLM descriptions with geometric cues for LLM reasoning, and the Dyn-THOR dynamic benchmark.

  • Results

    DSG consistently outperforms existing approaches in object node construction and spatial relationship reasoning on Dyn-THOR, 3RScan, and real-world scenes.

  • Takeaways & Limitations

    Comparing optimized and current viewpoints improves object updates, while combining visual context with geometry enables richer spatial relationships and improves coarse-grained recognition.

Abstract

from arXiv · show

In indoor environments, object positions frequently change due to human activities or embodied-agent interactions, causing previously constructed scene graphs to become inconsistent with the current scene. To address this issue, we propose DSG, a dynamic 3D scene graph construction framework that detects object changes and performs spatial relationship reasoning. First, we construct a semantic-aware 3D Gaussian scene representation and develop a dual-view rendering-based object change detection method to enable reliable scene graph node updates. Second, we propose a spatial relationship reasoning method that incorporates multi-granularity visual context, enabling a large language model to identify a richer set of interobject spatial relationships. Furthermore, we introduce DynTHOR, a dynamic indoor scene graph benchmark built on the AI2-THOR simulation platform for evaluating scene graph construction in dynamic environments. Extensive experiments on Dyn-THOR, 3RScan, and real-world scenes demonstrate that DSG consistently outperforms existing methods in both object node construction and spatial relationship reasoning, significantly improving the accuracy of dynamic scene graph construction.

I. INTRODUCTION

DSG addresses scene-graph inconsistency in changing indoor environments by updating object nodes through dual-view rendering and enriching spatial reasoning with visual context. It also introduces Dyn-THOR for quantitative evaluation and reports improvements across simulated and real-world settings.

  • Motivation: Changing object positions can invalidate previously constructed scene graphs used for embodied-agent perception, navigation, and manipulation.The motivation concerns maintaining scene-graph consistency as humans or robots alter indoor environments.
  • Framework: DSG constructs a semantic 3D Gaussian map and detects object changes through rendering comparisons from current and optimized viewpoints.The optimized-view comparison is designed to reduce false deletions caused by rendering artifacts and viewpoint limitations.
  • Framework: Visual-context-enhanced reasoning combines object-level and surrounding-region descriptions with geometric information to guide LLM inference of inter-object spatial relationships.The method uses VLM-generated multi-granularity descriptions together with object locations and 3D bounding boxes.
  • Benchmark: Dyn-THOR provides object-relocation scenarios and global ground-truth annotations for evaluating dynamic indoor scene-graph construction.The benchmark is built on AI2-THOR and targets scene-graph consistency after object relocation.
  • Results: Experiments on Dyn-THOR, 3RScan, and real-world scenes show consistent improvements in object node construction and spatial relationship reasoning.The reported evaluation covers both simulated and real-world environments.

II. RELATED WORK

Prior work has advanced high-fidelity 3D representations and multimodal scene-graph reasoning, but dynamic online maintenance and observation-specific visual context remain insufficiently addressed. Existing benchmarks also provide limited quantitative support for evaluating object relocation.

  • 3D Scene Representations: Earlier embodied-agent maps used voxels, point clouds, meshes, or ESDFs, trading off memory consumption, reconstruction quality, and object fidelity.3D Gaussian Splatting later offered high rendering quality and real-time rendering for online mapping.
  • 3D Scene Graph Construction: A 3D scene graph represents environments with object nodes and relationship edges, supporting structured embodied perception and decision making.Prior construction methods assign semantic and geometric attributes to object nodes and generate relationships from reconstructed scenes or RGB-D observations.
  • Multimodal Reasoning: VLM- and LLM-based methods improve spatial reasoning using geometric overlap, semantic labels, textual descriptions, contextualized visual features, or key-frame descriptions.These approaches extend scene-graph reasoning beyond purely geometric representations.
  • Open Challenges: Existing approaches insufficiently exploit observation-specific visual context between object pairs, limiting distinction between ambiguous relationships in complex indoor environments.The stated limitation concerns both contextual reasoning and the assumption of static environments.
  • Open Challenges: Most benchmarks target static environments and provide limited support for evaluating scene-graph construction after object relocation.The lack of quantitative object-change annotations makes robustness difficult to assess comprehensively.

III. METHOD

DSG builds object-centric semantic Gaussian representations by associating open-vocabulary detections with Gaussian primitives, then updates those objects through similarity-based association. The resulting representations support high-fidelity object modeling for dynamic scene graphs.

  • Object Representation: DSG incrementally represents each object with an independent set of semantic 3D Gaussian primitives.The map is constructed from posed RGB-D sequences.
  • Object Representation: GroundingDINO detects object boxes, SAM generates instance masks, and CLIP extracts instance-level visual features for semantic association.These open-vocabulary models connect image observations to objects in the global Gaussian map.
  • Object Association: Object association combines geometric mask similarity and semantic feature similarity into an overall matching score.The combined score supports greedy association between current detections and existing Gaussian objects.
  • Object Association: Successful matches update global semantic features by weighted averaging, while unmatched detections initialize new Gaussian objects.The weighting uses the number of observations associated with the existing Gaussian object.
  • Object Representation: Differentiable rendering optimizes Gaussian position, color, scale, rotation, and opacity to produce high-fidelity object representations.Each object remains represented by Gaussian primitives sharing an object identifier.

B. Object Change Detection

DSG detects object changes through optimized rendering from both the current and an optimized viewpoint. The optimized viewpoint improves visibility coverage while visibility regularization prevents excessive camera distance.

  • B. Object Change Detection: The method compares historical and current Gaussian maps from the current viewpoint and an optimized viewpoint to classify object changes.Objects are assigned Stable, Appeared, or Missing states after comparison and association.
  • B. Object Change Detection: Optimizing Gaussian primitives before rendering reduces artifacts caused by limited viewpoints and complex materials, improving object comparison reliability.This design reduces false deletions of valid objects caused by inaccurate rendered appearances.
  • B. Object Change Detection: The optimized viewpoint maximizes visibility of uncertain objects by keeping them within the camera frustum and near the image center.The frustum loss encourages complete observation through projected Gaussian points and image-coordinate constraints.
  • B. Object Change Detection: Visibility regularization prevents the optimized viewpoint from drifting excessively far from target objects.The visibility loss encourages target objects to occupy a larger image region while maintaining an appropriate observation distance.

C. Spatial Relationship Reasoning

DSG combines multi-granularity visual descriptions with geometric information to improve spatial relationship reasoning. It uses object and context descriptions alongside filtered candidate pairs to infer scene-graph edges.

  • C. Spatial Relationship Reasoning: The method combines multi-granularity visual descriptions with an LLM to infer spatial relationships between objects.The approach addresses limitations of reasoning based only on object labels and geometric information.
  • C. Spatial Relationship Reasoning: Object-level descriptions capture appearance and intrinsic attributes, while context-level descriptions include neighboring objects and surrounding spatial context.The context region is produced by expanding each object mask by a fixed factor of 2.5.
  • C. Spatial Relationship Reasoning: Candidate object pairs are filtered by 3D bounding-box IoU and point-cloud overlap before the LLM jointly uses geometry, labels, and visual descriptions to assign edge labels.The inferred relationship labels form the scene-graph edges together with updated Gaussian object nodes.

A. Datasets

The evaluation uses Dyn-THOR for controlled dynamic indoor scenes, 3RScan for real-world relocation scenarios, and a self-collected RGB-D dataset for further validation.

  • Experiments use Dyn-THOR, 3RScan, and a self-collected real-world dataset.These datasets cover simulated dynamic scenes, real-world rescans, and diverse indoor environments.
  • Dyn-THOR contains four indoor scene categories, 20 sequences, and 3,326 keyframes with synchronized RGB, depth, and camera-pose data.Its sequences compare unchanged scenes with stages where up to ten movable objects are relocated.
  • Dyn-THOR provides ground-truth states for Stable, Appeared, and Missing objects, plus 3D bounding boxes and spatial relationships.These annotations support quantitative evaluation of dynamic scene graph construction.
  • The 3RScan evaluation selects five reference–rescan pairs from 1,482 indoor scans, totaling 1,866 keyframes.Reference and rescan captures occur at different time instances, supporting evaluation under object relocation.
  • The self-collected real-world dataset contains 831 RGB-D frames covering diverse indoor objects and layouts.Images have 640×480 resolution, and camera poses are estimated using VINS-Fusion.

B. Experimental Setup

Node evaluation matches predicted and ground-truth objects using geometry and semantic similarity, while edge evaluation measures relationship completeness and correctness under fixed preprocessing settings.

  • Node evaluation matches objects using 3D IoU and the Hungarian algorithm, then verifies semantic consistency with CLIP similarity.Precision, Recall, and F1 are computed from the remaining valid matches.
  • Edge evaluation first counts evaluable edges, requiring both endpoint objects to be successfully matched.This separates graph completeness from relationship prediction quality.
  • Coarse Matches require a relationship between an object pair, whereas Fine Matches additionally require the subject, predicate, and object to match ground truth.
  • Evaluation preprocessing uses 0.01 m voxel downsampling, DBSCAN denoising, and confidence thresholds τhigh = 0.9 and τlow = 0.3.Node matching uses IoU and CLIP thresholds of 0.3 and 0.75, respectively.

C. Node-Level Evaluation

The proposed method achieves stronger node construction after object relocation than ConceptGraphs and DynamicGSG, with dual-view detection reducing residual and erroneous object representations.

  • Precision, Recall, and F1 consistently outperform existing baselines on both Dyn-THOR and 3RScan.
  • 2.1% MRR is achieved by the full dual-view variant on Dyn-THOR, while retaining Precision, Recall, and F1 comparable to the single-view variant.
  • ConceptGraphs retains obsolete objects after relocation, whereas DynamicGSG exhibits missed detections and erroneous deletions in qualitative Dyn-THOR comparisons.
  • On 3RScan, DynamicGSG frequently deletes valid objects under challenging real-world conditions, while the proposed method preserves more complete representations.
  • Optimizing Gaussian primitives before comparison and verifying uncertain objects from an optimized virtual viewpoint improves missing-object detection under incomplete observations.
  • Removing object culling raises MRR from 2.1 to 9.3, showing that Gaussian-level pruning alone is insufficient for reliable object matching.

D. Edge-Level Evaluation

The proposed method constructs denser and more accurate spatial relationships by combining object updates, geometric information, and multi-granularity visual context.

  • 15.5 edges per scene are predicted on average, providing richer relational information while maintaining high-quality object nodes.
  • Coarse matching reaches approximately 43%, compared with 22% for DynamicGSG and 8% for ConceptGraphs.The proposed method also achieves the highest number of fine-grained matches.
  • Adding visual context increases predicted valid edges from 13.6 to 15.5 and coarse-grained relation accuracy from 1.4 to 1.8.Fine-grained matching accuracy remains 0.8, while evaluable edges remain unchanged.
  • The proposed method combines accurate object updates with complementary object-level and surrounding-region descriptions, helping identify relationships missed by geometric cues alone.
  • In real indoor scenes, the method maintains accurate object representations despite challenging materials, incomplete observations, and previously unseen object categories.

V. CONCLUSIONS

DSG combines dual-view rendering comparison for object updates with visual-context-driven language-model reasoning for spatial relationships in changing indoor environments. Experiments in simulated and real-world scenes support more accurate updates, richer reasoning, and more complete scene graphs.

  • DSG combines dual-view rendering comparison for object updates with visual-context-driven language-model reasoning for spatial relationship inference.The framework targets indoor environments with changing object positions.
  • Optimized Gaussian renderings reduce false object deletions, while an optimized observation viewpoint identifies missing objects undetected from the current viewpoint alone.
  • Compared with DynamicGSG, DSG achieves more accurate object updates and richer relationship reasoning, producing a more complete and meaningful 3D scene graph.
  • Visual context with geometric information enables richer spatial relationships and improves coarse-grained relationship recognition.
Loading 2609.00619v1…