NoPA: Non-Parametric Online 3D Scene Graph Generation

• Only three prior works tackle online 3D SSG generation with real-time performance; SLAM-based methods incur computational overhead, while FROSS avoids explicit mapping using Gaussian objects.Source: §1 · pp. 1–2
• Gaussian ellipsoids discard geometric detail and make merging fragile, causing under-merging or incorrect merges across viewpoints that progressively degrade scene graphs.Thin or planar objects can produce near-singular covariances, while viewpoint changes can yield inconsistent covariances and spatial offsets.Source: §1 · pp. 2–3, Figure 1 · p. 2
• NoPA represents each object with a fixed-size non-parametric particle set, preserving geometric support while keeping memory and runtime complexity constant.It estimates a kernel density over the unified particle support of two candidates and resamples a fixed-size set to integrate multi-view evidence.Source: §1 · p. 3
• NoPA uses Maximum Mean Discrepancy to compare particle distributions for robust merging under viewpoint variation and noisy predictions, then propagates relations across similar-object clusters to recover missing edges.Source: §1 · p. 3
• NoPA achieves state-of-the-art performance on multiple online 3D SSG benchmarks while maintaining competitive real-time efficiency.Source: §1 · p. 3

Offline 3D SSG

• Offline 3D SSG estimates scene graphs from ground-truth geometry or multi-view RGB-D images, typically with non-incremental processing.Modern methods use techniques such as model-prior distillation, confidence rescoring, segmentation, and VLM/LLM refinement, but fixed-frame aggregation and expensive global association challenge online latency and bounded memory.Source: §2 · pp. 3–4
• NoPA replaces Gaussian object models with fixed-size particle sets, merges candidates using distribution-level criteria, and propagates relations within affinity clusters to recover missed edges during incremental fusion.The design targets constant memory per object while improving cross-view association.Source: §2 · p. 4

Online 3D SSG

• NoPA retains richer 3D geometry than FROSS while maintaining real-time performance, improving object merging and overall model performance.FROSS accelerates inference with 3D Gaussians but loses precise localization and relies on an approximation of 2D object shape from RGB-D observations.Source: §2 · p. 4
• Earlier online methods include Kim et al., whose local graphs miss real-time speeds; FAN uses dense 3D representations, while MonoSSG lifts 2D entities into 3D.MonoSSG also enhances ORBSLAM and uses a geometric gate to fuse geometric information with multi-view image features.Source: §2 · p. 4

• Given N multi-view RGB images of a 3D scene, the goal is to estimate its 3D scene graph.Source: §3 · p. 4
• The graph represents object categories as nodes and directed relationships as predicate-labeled edges, equivalently forming object-relation-object triplets.Source: §3 · p. 5
• At test time, the online method receives sequential partial observations rather than the full image set and incrementally updates the graph as images arrive during exploration.Source: §3 · p. 5

4 Preliminaries

• FROSS predicts a 2D scene graph from each RGB frame, lifts its nodes using depth and camera pose, then associates, merges, and updates global 3D objects.The lifting back-projects image evidence with the depth map and transforms it into the world frame.Source: §4 · p. 5
• FROSS represents each 3D object as a single Gaussian initialized from a lifted 2D bounding-box Gaussian, merging hypotheses with matching semantic labels when their Hellinger distance is below threshold δH.The Gaussian is parameterized by the object centroid and covariance.Source: §4 · p. 5
• The pipeline overview samples pixels within 2D boxes to form world-frame particle sets, uses a constant-time Hellinger pre-filter and MMD for ambiguous associations, and propagates relations within high-affinity clusters.The relation propagation targets dropouts from the 2D predictor.Source: Figure 2 · p. 6
• FROSS’s single-Gaussian approximation removes fine object structure and accumulates geometric error across frames, causing disjoint instances, fragile merging, and incorrect associations.Source: §4 · p. 6

5 Our Method

• The framework lifts local 2D scene graphs into 3D candidates, fuses them into the global graph, and propagates relations within affinity clusters while maintaining constant memory per object.It uses depth maps and camera poses for lifting, then stabilizes the relation set to recover relations missed by the 2D predictor in a single view.Source: §5 · p. 6

5.1 Non-Parametric Object Representation

• Each detected 2D box is lifted by depth back-projection and camera pose into a 3D particle set, preserving object support and multimodality under partial observations.Unlike a single Gaussian’s ellipsoidal prior, the particle representation reduces over-merging and under-merging across partial views.Source: §5.1 · pp. 6–7

5.2 Online Association and Merging

• A constant-time Hellinger pre-filter accepts clear matches or rejects clear mismatches, while MMD compares KDE distributions to resolve borderline candidate pairs.The margin band defers uncertain Hellinger cases to MMD, which detects support differences beyond first- and second-order moments.Source: §5.2 · p. 8
• After a merge, NoPA resamples the union of particle supports to a fixed-size set, preserving both candidates’ geometric information without particle growth over time.The fixed particle count maintains constant memory and avoids collapsing the representation toward a single mode during long-horizon fusion.Source: §5.2 · p. 9

5.3 Relationship Propagation with Affinity Clusters

• NoPA reuses MMD scores to cluster geometrically similar candidates, then propagates candidate relations within each cluster and finalizes relation types by majority vote.This recovers relations missed in a single view while reducing sensitivity to confidence thresholds and limiting relation drift.Source: §5.3 · pp. 9–10

6.1 Experimental Setup

• Experiments evaluate 3DSSG and ReplicaSSG, comparing against online RGB-D methods on 3DSSG and FROSS alone on ReplicaSSG.The reported metrics include top-1 recall for object, predicate, and relationship estimation; object and predicate mean recall; latency; and memory requirements.Source: §6.1 · pp. 10–11

6.2 Quantitative Results

• NoPA surpasses all baselines in 3DSSG top-1 recall and object mRecall with competitive latency and comparable VRAM; on ReplicaSSG, it beats FROSS across metrics, including a 65.5% relationship-recall increase.NoPA with 128 particles already surpasses all baselines on 3DSSG.Source: Table 1 · p. 11, §6.2 · pp. 11–12, Table 2 · p. 12

6.3 Qualitative Results

• NoPA better distinguishes thin-structure windows and correctly associates partial wall observations, while both methods struggle to distinguish a counter from a cabinet.FROSS often incorrectly merges partial wall observations; NoPA’s counter misclassification is also noted. Fig. 5 shows the kitchen scene comparison.Source: §6.3 · p. 12, Figure 5 · p. 13

6.4 Ablation Studies

• Replacing Gaussians with particle sets raises object recall but lowers predicate and relationship recall; tailored merging improves all metrics, and relationship propagation further improves predicate and relationship recall without reducing object recall.The method’s performance depends heavily on pretrained 2D scene-graph predictions, especially object predictions, which cap its upper bound.Source: §6.4 · pp. 12, 14, Table 3 · p. 14
• A lower MMD threshold favors object recall, while a higher threshold favors predicate and relationship recall; MMD also better captures distribution differences than Gaussian-based Hellinger distance in ambiguous cases.Relaxing merging can create larger merged objects that fail overlap-based matching; covariance alone may miss class-distribution differences from misclassified 2D predictions.Source: §6.4 · pp. 13–14, Table 4 · p. 14

• NoPA achieves state-of-the-art performance on multiple online 3D scene graph benchmarks while maintaining competitive real-time efficiency.Source: §7 · p. 14
• Fixed-size particle sets preserve geometric support with constant memory and runtime complexity, removing the restrictive ellipsoidal assumption of Gaussian object models.Source: §7 · p. 14
• MMD compares particle sets directly in feature space for robust merging under viewpoint variation and noisy predictions, while a Hellinger-distance pre-filter avoids unnecessary evaluations.Source: §7 · p. 14
• Relationship propagation recovers missing relations and improves global graph consistency.Source: §7 · p. 14

Generation Supplementary Material

• NoPA is evaluated in additional quantitative and qualitative experiments.The supplementary material also includes analyses of representation choice, relationship recovery, ambiguity, and the speed–performance balance.Source: Generation Supplementary Material · p. 19

A Additional Experiments

• Per-class experiments provide an additional evaluation of NoPA.These are listed as part of Sec. A.1.Source: Generation Supplementary Material · p. 19
• Experiments using ground-truth 2D SSG as input provide another evaluation setting.These are listed in Sec. A.2.Source: Generation Supplementary Material · p. 19
• Additional qualitative results are presented in Sec. A.3.Source: Generation Supplementary Material · p. 19

Generation Supplementary Material

• Sec. B.1 compares NoPA’s non-parametric object formulation with other parametric methods to justify the representation choice.Source: Generation Supplementary Material · p. 19
• Sec. B.2 contrasts relationship propagation with an alternate merging-based mechanism for recovering relationships lost to under-merging.Source: Generation Supplementary Material · p. 19
• Section B.2 contrasts relationship propagation with an alternate merging-based mechanism as approaches for recovering relationships lost through under-merging.Source: Generation Supplementary Material · p. 19
• Sec. B.4 examines how the number of particles per object affects NoPA’s performance and speed–performance balance.Source: Generation Supplementary Material · p. 19

A.1 Per-class Experiments

• FROSS and NoPA show similarly poor performance on rare predicates, a limitation attributed to severe 3DSSG class imbalance and their shared RT-DETR-EGTR 2D SSG backbone.The authors leave addressing class imbalance to future work because it is not this paper’s focus.Source: §A.1 · p. 20
• NoPA achieves the best performance in all but three 3DSSG object classes, outperforming FROSS and JointSSG, especially on picture, window, floor, and wall.The authors attribute FROSS’ poorer floor and wall performance to subpar merging, while NoPA’s merging yields objects with greater ground-truth overlap.Source: §A.1 · p. 19, Table 5 · p. 20
• NoPA significantly outperforms competing methods on attached to and excels on standing on relationships involving large furniture such as desks and cabinets.These gains occur for relationships associated with objects that NoPA’s merging correctly identifies, including wall for attached to.Source: Table 6 · p. 20, §A.1 · p. 20

A.2 Ground Truth 2D SSG Experiments

• NoPA depends heavily on pretrained 2D SSG prediction accuracy, particularly for object predictions.Source: §A.2 · p. 20
• Without accurate object classification in at least one frame, predicates involving a misclassified object can be wrong or missed, degrading the associated predicate and object classes.Source: §A.2 · p. 21
• With ground-truth 2D SSG inputs, NoPA improves across all metrics, especially predicate mRecall, and is more robust than FROSS to inferior 2D SSG quality.The results suggest NoPA performance scales with input accuracy.Source: §A.2 · p. 21, Table 9 · p. 21

A.3 More Qualitative Results

• Both methods often miss hanging on because their shared RT-DETR-EGTR predictions are constrained by the initial 2D scene-graph predictions; NoPA generalizes across room types, appearances, and lighting.The initial 2D predictions contain few correct instances of hanging on, limiting both methods’ final predicate predictions.Source: §A.3 · pp. 21–22
• Across three 3DSSG scenes, NoPA classifies objects and predicts most predicate relationships more accurately than FROSS, which often misses wall instances and relations.The examples also show FROSS confusing walls with sinks or other furniture, and sofas with chairs; NoPA correctly classifies most or all objects in the depicted scenes.Source: §A.3 · p. 21, Figure 6 · p. 22, Figure 7 · p. 23

B.1 Comparison with Other Representations

• Sparse lifted point clouds grow linearly with the number of frames, limiting real-time performance, and their overlap-based merging remains vulnerable to outlier points and incorrect extents.The implementation merges candidates when nearest-neighbor overlap exceeds 0.1; unlike bounding boxes, lifted clouds largely contain foreground objects, reducing background-space fusion.Source: §B.1 · pp. 23–24
• NoPA’s kernel-density resampling tends to remove particles far from the others, making its non-parametric representation less susceptible to outlier-driven object extents than boxes or point clouds.This helps keep estimated object extents closer to the ground-truth object extent.Source: §B.1 · p. 24

B.2 Analysis on Relationship Propagation

• Relationship propagation clusters same-class object candidates using affinity derived from precomputed MMD, then unions their relations to recover relations missed during exploration.It assumes neighboring candidates of the same class have similar relations; candidates below affinity threshold τ are not clustered, limiting oversized clusters and improving amortized computational efficiency.Source: §B.2 · pp. 24–25

B.1 Comparison with Other Representations

• Bounding-box merging uses an IoU threshold of 0.1, but outlier-expanded boxes and captured background can cause incorrect fusion; filtering oversized boxes removes relationships and reduces recall.The threshold rule merges candidates when their IoU exceeds the threshold and otherwise spawns a new global object; filtering can remove the accumulated relationships associated with oversized boxes.Source: §B.1 · pp. 22–23, Table 10 · p. 25

B.2 Analysis on Relationship Propagation

• Unlike multi-step label propagation, NoPA performs a single-step update of relations between already-labeled nodes; unlike merging, it preserves candidates, avoiding relation loss from incorrect merges.Incorrectly merging distinct instances can remove ground-truth matches, including after filtering, so previously augmented relations may still be lost and all metrics can degrade.Source: §B.2 · pp. 25–26

B.3 Analysis on Ambiguity

• Even a small number of incorrect merge decisions within the ambiguous margin band can cascade and degrade NoPA’s performance.The distribution of ambiguous merge decisions is shown for ReplicaSSG.Source: §B.3 · p. 26
• Table 11 compares relation aggregation methods on the 3DSSG test split.The supplied caption does not include the table’s values or identify a winning method.Source: Table 11 · p. 26
• NoPA performs best with ε = 0.05; ε = 0 uses only Hellinger distance, without additional MMD support.Table 12 compares margin-band values on the ReplicaSSG test split.Source: §B.3 · p. 27, Table 12 · p. 27
• MMD is most reliable in a narrow ambiguity band because it captures distributional distance but not the Euclidean distance encoded by Hellinger distance.A wider margin can lead MMD to merge candidates that are far apart in Euclidean space; clear-cut decisions are better left to Hellinger distance.Source: §B.3 · p. 27

B.4 Analysis on Number of Particles

• MMD’s Stage 2 runtime scales quadratically with particle count, but its use only for ambiguous decisions amortizes the cost, and runtime does not increase substantially as particles increase.The runtime comparison is reported in Table 13.Source: §B.4 · p. 27, Table 13 · p. 27
• At n = 256, NoPA strikes the right balance between speed and performance as the number of particles increases.The particle-count comparison is on the 3DSSG validation split.Source: Table 13 · p. 27, §B.4 · p. 28
• More particles do not guarantee better performance: NoPA with n = 512 performs worse than with n = 256.The passage suggests excess particles may duplicate information or produce noisier object representations.Source: §B.4 · p. 28