PaperlayerView exact revision on arXiv

Source-linked AI summary

NoPA: Non-Parametric Online 3D Scene Graph Generation

Qi Xun Yeo, Seungjun Lee, Yan Li, Gim Hee Lee

arXiv:2607.00529v1cs.CV

TL;DR

Prior 3D scene graph methods often operate offline, while Gaussian object models lose geometric detail and impair online merging. NoPA uses fixed-size non-parametric particle sets with MMD-based merging and achieves state-of-the-art performance on multiple online 3D scene graph benchmarks while retaining real-time complexity.

  • Problem

    Prior 3D scene graph generation largely uses offline settings, while single-Gaussian object models lose geometric detail and cause fragile merging and incorrect associations.

  • Method

    NoPA represents objects with fixed-size particle sets and uses MMD-based distribution merging plus similarity-guided relationship propagation.

  • Results

    NoPA achieves state-of-the-art performance on multiple online 3D scene graph benchmarks while preserving fixed memory usage and real-time computational complexity.

  • Takeaways & Limitations

    NoPA preserves richer geometric support and improves the stability of multi-view object associations without sacrificing real-time efficiency.

  • Takeaways & Limitations

    NoPA depends heavily on pretrained 2D scene graph predictions, whose object and relation accuracy caps its performance.

Abstract

from arXiv · show

Classic 3D scene graph generation approaches fail to work in real-time due to the heavy computational cost of environment mapping and the need to generate intermediate point-cloud representations. To alleviate this issue, a recent work eschews point clouds in favor of a lightweight Gaussian distribution for each object. This approximation drastically speeds up inference and enables real-time 3D scene graph generation. However, the representation has two key weaknesses. \textbf{1)} Each object is approximated by a single 3D Gaussian, which causes a severe loss of 3D geometric detail. \textbf{2)} The discrepancy between this approximation and the true object geometry exacerbates the inaccurate merging of object candidates during online inference. To address these issues, we propose \textbf{NoPA}, which represents each object as a separate non-parametric distribution. This formulation retains 3D geometric information while preserving real-time inference of the parametric Gaussian formulation. To build upon our novel object representation, we propose a tailored merging strategy to recover coherent object instances. Specifically, we leverage maximum mean discrepancy on kernel density estimates to enable robust merging of object candidates during online exploration while minimizing added computational complexity. The key is to maintain a fixed particle set per object. Furthermore, to rectify the relation loss caused by misclassified objects, NoPA propagates relationships between objects with high affinity. Experiments show that NoPA substantially outperforms current methods without sacrificing real-time inference speed.

1 Introduction

NoPA addresses the geometric and merging limitations of real-time online 3D scene graph generation by replacing single-Gaussian object models with fixed-size non-parametric particle sets. It combines distribution-level MMD merging and similarity-guided relationship propagation while preserving fixed memory, real-time complexity, and competitive performance.

  • Motivation: Prior online methods either incur SLAM-based reconstruction overhead or approximate each object as a Gaussian to achieve real-time performance.SceneGraphFusion and MonoSSG reconstruct geometry through SLAM, whereas FROSS avoids explicit mapping through Gaussian object modeling.
  • Motivation: FROSS’s ellipsoidal Gaussian assumption discards fine geometric structure, making merges fragile and causing accumulated incorrect merges and undermerges.Thin or planar structures such as pictures and windows are especially problematic, and the limitations stem from the parametric assumption.
  • NoPA representation: NoPA represents each object with a fixed-size non-parametric particle set that preserves geometric support while maintaining constant memory and runtime complexity.This removes the restrictive ellipsoidal assumption without reintroducing the computational burden of SLAM-based pipelines.
  • Merging: NoPA uses Maximum Mean Discrepancy to compare particle-set distributions directly, replacing unreliable covariance similarity and improving merging robustness under viewpoint variation and noisy predictions.The distribution-level criterion addresses under-merging caused by covariance comparisons across views.
  • Graph consistency: NoPA propagates relationships according to distribution similarity to recover missing relations and reinforce graph consistency.The paper reports state-of-the-art performance on multiple online 3D SSG benchmarks while maintaining competitive real-time efficiency.

2 Related Work

Offline 3D scene graph generation estimates scene graphs non-incrementally from ground-truth 3D geometry or multi-view RGB-D images. Early work modeled pairwise relationships, while later methods used pretrained model priors, confidence rescoring, and instance masks to improve predictions.

  • Offline 3D SSG: Offline 3D SSG methods estimate scene graphs non-incrementally from ground-truth 3D geometry or multi-view RGB-D images.Most modern approaches rely on multi-view RGB-D images.
  • Offline 3D SSG: Wald et al. first formulated 3D SSG generation by modeling pairwise relationships to predict the graph.
  • Offline 3D SSG: Wang et al. distilled knowledge from a multimodal oracle model into a 3D model to leverage pretrained model priors.
  • Offline 3D SSG: Yeo et al. refined low-confidence predictions through statistical confidence rescoring and enhanced node features using SegmentAnything instance masks.

3 Problem Definition

The task estimates a 3D scene graph from multi-view RGB images, representing objects as labeled nodes and directed relationships as labeled edges. In the online setting, the graph is incrementally updated from sequential partial observations during scene exploration.

  • Given N multi-view RGB images of a 3D scene, the goal is to estimate a 3D scene graph.
  • Each scene-graph node has an object category label, while each directed edge has a predicate label and forms a triplet (o_k, r_k→j, o_j).
  • At test time, the online method receives sequential partial observations rather than the full image set and incrementally updates G_3D during scene exploration.

4 Preliminaries

FROSS incrementally constructs a global 3D scene graph from RGB frames by lifting per-frame 2D predictions into the world frame and fusing them with existing objects. Its single-Gaussian object representation enables Hellinger-based association but loses geometric detail and can produce fragile, incorrect merges.

  • FROSS pipeline: FROSS predicts a per-frame 2D scene graph, lifts it using depth and camera pose, and fuses it into the global 3D scene graph.The framework avoids explicit environment mapping while updating the scene graph incrementally from RGB frames.
  • FROSS pipeline: Back-projection maps each 2D node to a 3D object hypothesis, while fusion associates lifted hypotheses with existing nodes before merging and updating states.The back-projection uses the per-frame depth map and camera pose to transform image evidence into the world frame.
  • Object representation: FROSS represents every 3D object node with a single Gaussian in R3, parameterized by its centroid and covariance.The Gaussian is initialized by lifting a 2D Gaussian estimated from the predicted 2D bounding box.
  • Data association: During fusion, matching semantic labels and a Hellinger distance below δH trigger merging between a lifted hypothesis and an existing global object.For Gaussian objects, the Hellinger distance has a closed form based on the Bhattacharyya distance.
  • Limitations: The single-Gaussian approximation imposes an ellipsoidal geometry that removes fine structural detail, accumulating errors that cause disjoint instances and incorrect associations.The loss of geometric detail makes online merging fragile across streaming images.

5 Our Method

NoPA replaces FROSS’s single-Gaussian object model with non-parametric particle sets and distributional online fusion. It combines a Hellinger pre-filter, MMD-based ambiguity resolution, fixed-size resampling, and affinity-based relation propagation.

  • Framework overview: NoPA lifts detected 2D objects into 3D particle sets using depth and camera pose, then fuses local candidates into the evolving global scene graph.A pretrained RT-DETR-EGTR model provides local 2D scene graphs from RGB frames.
  • Object representation: Particle sets preserve object support in R3 and permit multi-modality, reducing over-merging and under-merging caused by single-Gaussian ellipsoidal approximations.The representation treats particles as samples from an unknown occupancy distribution and forms a kernel density estimate with an RBF kernel.
  • Online association: NoPA first uses a constant-time Hellinger pre-filter for clear matches or mismatches, then applies MMD between KDEs only to candidates inside the margin band.The Hellinger margin band handles unstable decisions under depth noise, truncation, or limited view overlap, while MMD detects support mismatch beyond moment matching.
  • Online association: After merging, NoPA unions candidate particles and resamples a fixed-size set of n particles, preserving geometric information while preventing particle growth and maintaining constant memory.Resampling also avoids collapsing the representation toward a single mode during long-horizon fusion.
  • Relation propagation: NoPA builds affinity clusters from ambiguous-pair MMD scores, propagates candidate relations among geometrically similar objects, and uses majority voting to recover missed edges and reduce false positives.Observed 2D relations are first copied to the corresponding 3D node before cluster-based propagation.

6 Experiments

NoPA is evaluated on 3DSSG and ReplicaSSG against online 3D scene graph generation baselines using recall, mean recall, latency, and VRAM metrics. It surpasses prior methods while retaining real-time efficiency, and ablations support its non-parametric representation, MMD-based merging, and relationship propagation.

  • Datasets and baselines: Experiments use 3DSSG and ReplicaSSG, comparing against Kim’s framework, JointSSG, and FROSS on 3DSSG and FROSS on ReplicaSSG.3DSSG contains 1482 scenes, while ReplicaSSG contains 18 scenes.
  • Main results: NoPA surpasses all baselines in top-1 recall and object mRecall while maintaining competitive latency and comparable VRAM usage for real-time inference.With 128 particles, NoPA already surpasses all baselines; on ReplicaSSG, it surpasses FROSS across all metrics.
  • Qualitative results: Qualitatively, NoPA more reliably differentiates thin-structured objects and correctly associates partial wall observations, while counter-versus-cabinet ambiguity remains difficult.The comparison is shown against FROSS on a kitchen scene from 3DSSG.
  • Ablation study: Ablations show that particle-set representations improve object recall, while FROSS merging degrades performance on the non-parametric representation and relationship propagation further contributes to performance.The ablation isolates the non-parametric particle set, MMD-based merging, and relationship propagation components.
  • Merging analysis: Lower MMD thresholds improve object recall at the expense of predicate recall, whereas higher thresholds improve predicate and relationship recall while trading off object recall.The threshold analysis varies δ_MMD from 0.6 to 0.9 and attributes the trade-off to stricter merging criteria.
  • Limitations: NoPA’s performance remains bounded by the accuracy of pretrained 2D scene graph predictions, particularly object predictions, detections, and relations.The quality of 2D detections and relations caps the upper bound of the 3D approach.

7 Conclusion

NoPA presents a non-parametric framework for online 3D scene graph generation from multi-view RGB-D observations. It replaces Gaussian object models with fixed-size particle sets and introduces MMD-based distribution-level merging to preserve geometric support and stabilize multi-view associations.

  • NoPA generates online 3D scene graphs from multi-view RGB-D observations using a non-parametric framework.
  • Fixed-size particle sets replace Gaussian object models while preserving geometric support with constant memory and runtime complexity.
  • This representation removes the restrictive ellipsoidal assumption and yields more stable multi-view object associations.
  • NoPA introduces a distribution-level merging criterion based on MMD that compares particle sets directly.

NoPA: Non-Parametric Online 3D Scene Graph · A Additional Experiments

The supplementary material presents additional qualitative and quantitative evaluations of NoPA, including per-class, ground-truth-input, and qualitative experiments. It also analyzes representation choice, relationship propagation, ambiguity margins, and particle-count trade-offs affecting performance and speed.

  • NoPA: Non-Parametric Online 3D Scene Graph: The supplementary material conducts more qualitative and quantitative analysis of NoPA beyond the main evaluation.The additional analyses are presented as supplementary material.
  • NoPA: Non-Parametric Online 3D Scene Graph: Additional experiments evaluate NoPA quantitatively and qualitatively through per-class analysis, ground-truth 2D SSG inputs, and further qualitative results.These studies are organized in Sections A.1–A.3.
  • NoPA: Non-Parametric Online 3D Scene Graph: Section B.1 compares NoPA’s non-parametric object representation with other parametric methods to justify the chosen representation type.The analysis focuses on the effectiveness of the non-parametric formulation.
  • NoPA: Non-Parametric Online 3D Scene Graph: Section B.2 contrasts relationship propagation with an alternate merging mechanism for recovering relationships lost through under-merging.The comparison is intended to explain why relationship propagation works.
  • NoPA: Non-Parametric Online 3D Scene Graph: Section B.3 varies the margin band to assess how the definition of ambiguity affects NoPA’s performance and its emphasis on hard merge decisions.The experiment explains the choice to focus on hard merge decisions.
  • NoPA: Non-Parametric Online 3D Scene Graph: Section B.4 studies how the number of particles representing each object influences NoPA’s performance, clarifying the speed–performance balance.The particle-count analysis seeks an appropriate balance between inference speed and performance.

A.1 Per-class Experiments

On the 3DSSG test split, NoPA achieves the best performance across nearly all object classes and improves recognition of relationships involving correctly merged objects. Predicate performance remains poor on rare classes because of severe class imbalance inherited from the shared 2D backbone.

  • Object classes: NoPA achieves the best performance in all but three object classes on the 3DSSG test split.The per-class comparison is reported against reproduced FROSS and JointSSG results.
  • Object classes: NoPA better differentiates thin planar objects, including pictures and windows, than FROSS and JointSSG.
  • Predicate classes: Both FROSS and NoPA show similarly poor performance on rare predicate classes because severe class imbalance affects their shared RT-DETR-EGTR backbone.The imbalance is inherited from 2D scene-graph prediction.
  • Predicate classes: NoPA excels at classifying relationships co-occurring with objects that its merging formulation correctly identifies.
  • Predicate classes: NoPA significantly outperforms all competing methods on the attached to predicate class.

A.2 Ground Truth 2D SSG Experiments

NoPA’s performance depends on the quality of pretrained 2D scene graph predictions, particularly object classification. Ground-truth 2D inputs substantially improve NoPA across metrics, while NoPA remains more robust to inferior predictions than FROSS.

  • NoPA heavily depends on accurate pretrained 2D SSG predictions, especially object predictions.
  • Misclassified objects make their associated predicates incorrect or missed, degrading both predicate and affected object-class performance.FROSS performs poorly on the attached to predicate, which is commonly associated with the frequently misclassified wall class.
  • Ground-truth 2D SSG inputs significantly improve NoPA across all metrics, especially predicate mRecall.The comparison uses oracle variants of FROSS and NoPA that take ground-truth 2D SSG as input.
  • NoPA’s performance scales with more accurate 2D SSG inputs and is more robust to inferior 2D SSG quality than FROSS.

A.3 More Qualitative Results … B.2 Analysis on Relationship Propagation

NoPA improves qualitative object and relationship predictions across diverse 3DSSG scenes, while its non-parametric representation avoids key limitations of bounding-box and sparse-point-cloud alternatives. Relationship propagation recovers missed relations using affinity-based clustering, but incorrect merges can still degrade performance.

  • A.3 More Qualitative Results: NoPA more reliably classifies objects and predicts predicate relationships than FROSS across qualitative 3DSSG scenes.FROSS frequently misses wall instances, confuses wall and sink classes, and omits relationships; NoPA correctly classifies most or all objects and predicts most relationships in the shown scenes.
  • A.3 More Qualitative Results: NoPA remains robust across room types, object appearances, and lighting conditions.The qualitative scenes correspond to different room types, supporting generalization across diverse environments.
  • B.1 Comparison with Other Representations: Bounding-box merging can expand beyond true object extents because outliers and background regions inflate overlap.The implementation uses IoU with a hard threshold of δ_IoU = 0.1, but expanded boxes may still fuse objects whose overlap primarily represents background.
  • B.1 Comparison with Other Representations: Sparse point-cloud representations do not scale to real time when objects appear in many frames because point counts grow linearly.Their overlap is calculated using nearest neighbors with a hard threshold of δ_overlap = 0.1, while foreground-focused lifting largely avoids background space.
  • B.1 Comparison with Other Representations: NoPA’s resampling after kernel density estimation reduces outlier influence and limits object extent expansion beyond the ground-truth geometry.This distinguishes the non-parametric representation from bounding boxes, which lack a mechanism to shrink after expansion.
  • B.2 Analysis on Relationship Propagation: Relationship propagation recovers relations missed during incremental exploration when they were observed at some stage.It assumes neighboring candidates of the same class have similar relations with similar objects and reuses precomputed MMD for clustering.
  • B.2 Analysis on Relationship Propagation: Affinity filtering prevents low-affinity segments from sharing a cluster, avoiding overly large clusters and improving amortized computational efficiency.The relationship propagation mechanism is specified in Algorithm 1, which accumulates relations among objects within each cluster.
  • B.2 Analysis on Relationship Propagation: Incorrectly merging distinct object instances can remove ground-truth matches and cause augmented relations to be lost, degrading all metrics.Both matches may be removed when the combined instance is filtered by strict matching criteria.

B.3 Analysis on Ambiguity · B.4 Analysis on Number of Particles

NoPA’s ambiguity analysis shows that merge errors near the decision margin can cascade, motivating a narrow MMD-supported margin band. Particle-count analysis finds that n = 256 balances speed and performance, while larger sets can reduce accuracy without substantial runtime gains.

  • B.3 Analysis on Ambiguity: Ambiguous merge decisions in ReplicaSSG can cause cascading errors that degrade NoPA’s performance.Fig. 9 examines the distribution of merge decisions considered ambiguous.
  • B.3 Analysis on Ambiguity: The ambiguity study compares relation aggregation methods and margin-band values on the 3DSSG and ReplicaSSG test splits.Tables 11 and 12 report these comparisons, with best and second-best results highlighted where specified.
  • B.3 Analysis on Ambiguity: NoPA performs best with ε = 0.05, whereas ε = 0 uses only Hellinger distance without additional MMD support.The comparison varies margin bands corresponding to different ambiguity levels on ReplicaSSG.
  • B.3 Analysis on Ambiguity: MMD is effective for ambiguous merges but can misclassify clear-cut decisions because it omits the Euclidean distance encoded by Hellinger distance.Larger margin bands increase the likelihood of erroneous MMD-based decisions.
  • B.4 Analysis on Number of Particles: MMD runtime theoretically scales as O(n^2), but restricting it to ambiguous decisions amortizes the added cost.Empirically, increasing the particle count does not substantially increase NoPA’s runtime.
  • B.4 Analysis on Number of Particles: n = 256 strikes the right balance between speed and performance.Table 13 compares different particle counts on the 3DSSG validation split.
  • B.4 Analysis on Number of Particles: NoPA with n = 512 performs worse than n = 256, indicating that additional particles can replicate information or introduce noise.More particles do not necessarily improve performance and may become counterproductive.
Loading 2607.00529v1…