SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation

1. Introduction

• The motivating query is to predict plausible object types that match the surrounding context of an input 3D scene.For example, a speaker is more likely than a plant near a TV stand and room corner.Source: §1 · p. 1
• SceneGraphNet predicts a probability distribution over object types at query locations by passing messages through a dense graph of scene objects and their relationships.The graph captures spatial and structural relationships, and the method uses attention to weight messages.Source: §1 · p. 1, Figure 1 · p. 1
• SceneGraphNet predictions can be fused with multi-view CNN predictions to improve context-based object recognition.The paper also presents progressive scene population using objects placed at their most likely predicted locations.Source: §1 · p. 1, Figure 2 · p. 2, Figure 3 · p. 2
• 16% (51% →67%) improvement over the previous best method for predicting missing object types in SUNCG-based experiments.The evaluation intentionally left object types out of scenes.Source: §1 · p. 2
• The architecture models both short- and long-range object relationships, with attention-weighted iterative message passing for object-related prediction tasks.The listed tasks include spatial query answering, context-based recognition, and iterative scene synthesis.Source: §1 · pp. 1–2

2. Related Work

• Earlier indoor scene synthesis methods used hand-engineered kernels and graph walks or probabilistic models of object co-occurrence, placement, and compatibility.These approaches included synthesis from sketches and RGBD input.Source: §2 · p. 2
• Earlier methods were mostly limited to small scenes, with generalization constrained by coarse relationships, shape representations, and shallow or hand-engineered statistics.Larger scene datasets such as SUNCG enabled more sophisticated learning approaches.Source: §2 · p. 2
• Learning-based scene synthesis included image-based CNNs, a VAE–GAN, and GRAINS, which generates tree-structured scenes but directly encodes only local dependencies within groups.GRAINS also constructs its scene tree using hand-engineered heuristic grouping operations.Source: §2 · p. 2
• A concurrent graph method prunes spatial and semantic edges heuristically, whereas SceneGraphNet models dense graphs with short- and long-range dependencies and learns which relationships to attend to.SceneGraphNet also adapts message passing and aggregation to 3D indoor scene modeling.Source: §2 · p. 2

3.1. Message passing

• Each node starts from category, centroid, and scale features; relationship-specific messages are attention-weighted and aggregated with GRUs to update context-aware node representations.Attention weights are computed from raw shape representations; the updated node representations carry context into subsequent message-passing iterations.Source: §3.1 · pp. 3–4

3.2. Graph structure and Object Relationships

• Fine-grained structural relationships outperform coarse proximity links, while dense long-range co-occurrence interactions improve predictions despite added computation.Low-attention edges can be discarded at test time to accelerate message passing.Source: §3.2 · pp. 4–5
• The graph encodes directional support and supported-by edges, symmetry-based surrounding relations, and symmetric next-to relations between adjacent objects on the same supporting surface.Using separate message weights for the two support directions improved performance over symmetric messages; using only supporting edges performed worse.Source: §3.2 · p. 5

3.3. Prediction

• For a query, the method adds an empty node at that location, connects it by scene relationships, and decodes its representation after message passing into object-category probabilities.The implementation uses three message-passing iterations; training removes a random object other than a wall or floor and predicts its category.Source: §3.3 · pp. 5–6

3.4. Applications

• For scene recognition, multiplying and renormalizing the contextual distribution with a shape-based posterior improved predictions over multi-view object predictions alone.For incremental synthesis, the method evaluates category probabilities over candidate locations, selects the highest-probability category and location, and retrieves a model matching the predicted size; user supervision is eventually needed for placement, orientation, and stopping.Source: §3.4 · p. 6

Evaluation

• Evaluation used SUNCG bedrooms, living rooms, bathrooms, and offices, with one non-wall object removed and its centroid used as the query location.Rooms with layouts other than four-wall rectangles were excluded for fair comparisons; all methods used the same train, validation, and test split.Source: §4 · pp. 6–7

Results

• 10.2% (top-5) to 16.4% (top-1): SceneGraphNet outperformed competing methods in top-K accuracy averaged across room types.The reported margins are 10.2% for top-5 accuracy and 16.4% for top-1 accuracy.Source: Table 1 · p. 7, §4 · p. 7
• 26cm average size-prediction error for SceneGraphNet, compared with 38cm for GRAINS.Source: §4 · p. 7

Ablation Study

• SceneGraphNet outperformed degraded variants that changed graph structure, message aggregation, or message weighting.The tested variants included tree and sparse graphs, alternative aggregation schemes, and alternatives to learned attention weighting.Source: §4 · p. 7

Performance wrt iteration number

• Top-5 accuracy rose from 72.7% after the first message-passing iteration to 88.2% after the second and 89.5% after the third, then oscillated around 89%.Source: §4 · p. 7

Object recognition in 3D scenes

• 13%: combining SceneGraphNet’s context-aware posterior with MVCNN predictions improved object-recognition accuracy.Table 2 compares MVCNN alone with MVCNN combined with the contextual posterior.Source: Table 2 · p. 8, §4 · p. 8

• SceneGraphNet operates on a dense scene graph to support scene augmentation and context-based object recognition.Source: §5 · p. 8
• Figure 8 reports average top-3 classification accuracy for rooms binned by their number of objects.Source: Figure 8 · p. 8
• Predictions are limited to object categories at a given location; generating objects or whole scenes from scratch remains a possible extension.Source: §5 · p. 8
• Message updates are not guaranteed to converge and can oscillate, while matching training and testing iteration counts helps prevent instability.Source: §5 · p. 8

• Initialization, message, attention, aggregation, update, and prediction modules encode objects, weight pairwise messages, aggregate six relation channels, and output category probabilities or sizes.The initialization, message, aggregation, update, and prediction representations are 100-dimensional; attention outputs a scalar weight between 0 and 1.Source: Supplementary Materials · p. 10
• Supporting relations use a 0.05-meter height-distance threshold, while surrounding relations require symmetry within 0.2 meters and bounding-box sizes within a factor of 1.2.Source: Supplementary Materials · p. 10
• Figure 10 compares competing methods' top-K object-category accuracy for K values from 1 to 10.Source: Figure 10 · p. 10, Supplementary Materials · p. 11
• Experiments use SUNCG bedrooms, living rooms, bathrooms, and offices, with 51, 51, 31, and 42 categories respectively; Figure 9 summarizes room object counts.Source: Supplementary Materials · p. 10, Figure 9 · p. 10
• Living-room top-1 accuracy exceeds Wang et al. [20] by 21.0%, while bathroom top-1 accuracy is 8.4% higher.Table 3 reports top-K category-prediction accuracy by room type.Source: Supplementary Materials · p. 10, Table 3 · p. 11