PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences

Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, Federico Tombari

arXiv:2103.14898v3cs.CVcs.LG

TL;DR

The paper addresses online construction of semantic 3D scene graphs from partial RGB-D observations, a setting complicated by missing and changing graph information. SceneGraphFusion incrementally predicts and fuses graph semantics using segment features, a graph neural network, and feature-wise attention. It outperforms prior 3D scene graph methods, matches other 3D semantic and panoptic segmentation approaches, and runs at 35 Hz.

  • Problem

    Online semantic scene graph construction must estimate instances and relationships from partial, incomplete 3D geometry with changing graph neighborhoods.

  • Method

    SceneGraphFusion incrementally predicts and fuses semantic graphs from geometric segments using a graph network with feature-wise attention for missing graph information.

  • Results

    The method outperforms 3D scene graph prediction methods, achieves performance on par with 3D semantic and panoptic segmentation methods, and runs at 35Hz.

  • Takeaways & Limitations

    Incremental semantic scene graphs can be built during 3D mapping and may support future camera-pose retrieval or loop-closure detection in SLAM.

Abstract

from arXiv · show

Scene graphs are a compact and explicit representation successfully used in a variety of 2D scene understanding tasks. This work proposes a method to incrementally build up semantic scene graphs from a 3D environment given a sequence of RGB-D frames. To this end, we aggregate PointNet features from primitive scene components by means of a graph neural network. We also propose a novel attention mechanism well suited for partial and missing graph data present in such an incremental reconstruction scenario. Although our proposed method is designed to run on submaps of the scene, we show it also transfers to entire 3D scenes. Experiments show that our approach outperforms 3D scene graph prediction methods by a large margin and its accuracy is on par with other 3D semantic and panoptic segmentation methods while running at 35 Hz.

1. Introduction

The paper targets online semantic scene graph construction from RGB-D sequences, where prior 3D methods generally require complete geometry and operate offline. SceneGraphFusion incrementally builds globally consistent graphs while handling partial data and reports strong prediction and segmentation performance at real-time speed.

  • Prior 3D scene understanding methods typically process complete scans offline and require reconstructed 3D geometry beforehand.
  • Online semantic graph mapping must detect instances, estimate relationships, and cope with partial or incomplete 3D geometry.
  • SceneGraphFusion incrementally builds a globally consistent semantic scene graph alongside 3D mapping from RGB-D input.
  • Its inductive graph network handles missing nodes and edges while learning semantic labels, object instances, and same-part relationships.
  • 35Hz runtime accompanies performance that surpasses 3D scene graph prediction and is on par with 3D semantic and instance segmentation benchmarks.

2. Related Work

Related work includes offline 3D understanding from complete scans and incremental semantic SLAM from incoming RGB-D frames. SceneGraphFusion differs by focusing on semantic relationships and scene graphs during reconstruction without requiring prior complete-scene knowledge.

  • Complete-scan methods perform semantic or instance understanding from full 3D volumes or point clouds, whereas incremental SLAM operates directly on incoming RGB(-D) sequences.
  • Incremental semantic SLAM reconstructs scenes and extracts semantics in real time but often requires fusion or regularization for contradictory predictions and temporal consistency.
  • Semantic SLAM methods generally do not focus on semantic scene graphs or relationships between object instances.
  • Scene graph prediction extends instance segmentation by adding relationships between scene entities represented as graph nodes.
  • The framework uses efficient graph updates so unchanged lower-layer node features can be reused when one node changes.
  • Unlike prior 3D graph prediction, the proposed method needs no prior scene knowledge and predicts instances, semantics, and relationships while reconstruction proceeds.

3. Incremental 3D Scene Graph Framework

SceneGraphFusion combines incremental geometric reconstruction and segmentation with graph-based semantic prediction. It updates recently observed segments, predicts node and edge semantics, and fuses repeated predictions into a globally consistent graph.

  • The system has reconstruction-segmentation and scene graph prediction cores that process RGB-D frames with associated poses.
  • Each semantic graph node represents a segment with an object category, while edges encode predicates such as standing on and attached to.
  • Incremental geometric segmentation maintains a global 3D map whose segments can be added, merged, or removed as new frames arrive.
  • Segment properties include centroid, positional standard deviation, axis-aligned bounding-box dimensions, maximum length, and volume.
  • Neighbor edges connect segment pairs whose bounding-box distance is below the 0.5-meter proximity threshold used in experiments.
  • Only recently updated segments and their neighboring graph subset are processed, improving scalability and efficiency.
  • Repeated node and edge predictions are fused using running averages over stored class or predicate probabilities and weights.
  • Segment-level prediction preserves complete label-probability distributions with less memory than point-level methods.

4. Scene Graph Prediction

Scene graph prediction uses segment geometry, learned node and edge features, and graph message passing to classify segments and relationships. Feature-wise attention addresses changing neighborhoods and missing graph elements during incremental reconstruction.

  • Motivation: Changing segment shapes and neighbor counts make robust GNN feature learning difficult during incremental reconstruction.Segments are often incomplete and simple, while their neighborhoods change over time.
  • Feature-wise Attention: Feature-wise attention reweights individual latent features at each target node rather than distributing attention across edges.Max aggregation of the reweighted embedding reduces sensitivity to missing neighboring points; multi-head attention permits more flexible feature distributions.
  • Network Architecture: The network takes segment properties and a neighbor graph, then predicts a class for each segment and a predicate for each edge.Two MLP classifiers produce node-class and edge-predicate predictions, trained with joint cross-entropy losses.
  • Network Architecture: PointNet encodes each segment point cloud, which is concatenated with scale-related geometric properties to form its node feature.The properties include standard deviation, log bounding-box size, volume, and length.
  • Network Architecture: A two-layer GNN propagates node and edge features using neighborhood information, with MLP updates and feature-wise attention.Both node and edge features are updated in each message-passing layer.

5. Data Generation

The data-generation procedure introduces same-part relationships so segments belonging to one object can be clustered into instances. Evaluation uses complete 3D data with 160 objects and 26 predicate classes.

  • Relationship Generation: Same-part relationships are assigned to segments matched to the same ground-truth object, enabling instance-level segmentation from over-segmented maps.Ground-truth relationships on an object instance are inherited by all of its segments.
  • Evaluation: Table 1 evaluates scene graph prediction on 3RScan/3DSSG using complete 3D data, 160 objects, and 26 predicate classes.The evaluation separately concerns scene graph prediction under the stated object and predicate-class configuration.

6. Evaluation

The evaluation measures semantic scene graph prediction, semantic and panoptic segmentation, robustness to missing edges, and runtime. SceneGraphFusion improves graph prediction and feature-wise attention robustness, while achieving competitive segmentation performance at real-time speed.

  • 6.1. Semantic Scene Graph Prediction: SceneGraphFusion outperforms 3DSSG on relationship prediction by +0.45 R@50 and +0.21 R@100 using ground-truth instances.The improvement is attributed to small gains in predicate and object classification.
  • 6.1. Semantic Scene Graph Prediction: Feature-wise attention, FAT, gives a large-margin improvement over 3DSSG on geometric-segment graph prediction and outperforms GAT and SDPA.Incremental prediction is slightly worse than full-scene prediction, while the proposed fusion improves performance further.
  • 6.1. Semantic Scene Graph Prediction: Joint relationship prediction improves object classification compared with training without the predicate loss.This ablation evaluates graphs computed from the full 3D scene.
  • 6.2. 3D Panoptic/Semantic Segmentation: The method achieves the second-best mAP among compared incremental semantic segmentation methods while running at 35 Hz on a CPU.The comparison includes SemanticFusion, ProgressiveFusion, and FusionAware.
  • 6.2. 3D Panoptic/Semantic Segmentation: PanopticFusion has higher recognition quality, while SceneGraphFusion’s segmentation and panoptic quality are on par or slightly worse under missing-geometry evaluation.Skipping unreconstructed regions often improves SceneGraphFusion’s performance, but that evaluation is not directly comparable.
  • 6.3. Robustness and Runtime: FAT achieves average IoU scores of 49.3 with all edges and 41.9 after dropping 50% of edges, exceeding the other attention variants.The comparison includes no attention, SDPA, and GAT on segment classification.
  • 6.3. Robustness and Runtime: The system uses 28 ms for scene reconstruction and 133 ms for graph prediction on the reported CPU setup.Only two threads are used: one for reconstruction and one for graph prediction.

7. Conclusion

SceneGraphFusion incrementally fuses partial graph predictions from geometric segmentation into a globally consistent semantic map. FAT handles missing graph information effectively, while the resulting segmentation remains competitive at 35 Hz.

  • 7. Conclusion: SceneGraphFusion incrementally fuses partial graph predictions into a globally consistent semantic map.The method uses geometric segmentation as the source of partial graph predictions.
  • 7. Conclusion: FAT outperforms other attention mechanisms when handling missing graph information.The conclusion also reports competitive semantic and panoptic segmentation performance at 35 Hz.

8. Supplementary Material

The supplementary material details the network, training setup, attention comparison, runtime analysis, panoptic segmentation evaluation, and qualitative transfer across datasets. Results show robustness to missing edges, competitive panoptic segmentation quality, reduced computation through graph updates, and reasonable predictions across 3RScan and ScanNet.

  • Network Architecture: The PointNet encoder uses a shared-weight MLP followed by max pooling, while the GNN architecture is specified through fully connected layers and MLPs.The PointNet encoder is an MLP(64, 128, 512) followed by global max pooling.
  • Training Setup: Training uses randomly extracted subgraphs, 50% edge dropout, sampled segment points, and a 0.1 weighting factor between object and predicate losses.Subgraphs include two selected segments and their four-hop neighbors.
  • 3D Panoptic Segmentation: The method outperforms PanopticFusion in panoptic quality for 7 of 20 classes.Its segmentation quality is similar to PanopticFusion, while recognition quality is lower, likely because incremental segmentation can miss scene geometry.
  • Attention Comparison: The proposed attention method consistently outperforms alternative attention methods even when 50% of edges are dropped.Robustness varies by class: chairs, curtains, desks, floors, and walls are more robust, whereas bathtubs, beds, shower curtains, and windows depend more on neighboring information.
  • Runtime Analysis: Updating node and edge features with the graph structure significantly reduces computation time during incremental processing.Scene graph prediction runs in a separate thread and blocks the main thread only during data copying and fusion.
  • Qualitative Results: A model trained on 3RScan produces reasonable 3D scene graphs on ScanNet scenes despite the domain gap and lack of ScanNet relationship ground truth.The qualitative results include both 3RScan and ScanNet scenes.
Loading 2103.14898v3…