Source-linked AI summary
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, Silvio Savarese
TL;DR
The paper asks where diverse semantic information should be grounded and how it should be structured for useful, invariant scene understanding. It builds a 3D Scene Graph over buildings and a mainly automatic pipeline using framing and multi-view consistency, demonstrating the approach on Gibson.
Problem
The paper addresses how to ground diverse scene semantics—including object geometry, entity categories, and camera viewpoints—in a useful and invariant structure.
Method
The approach grounds semantics on a building’s 3D mesh in a four-layer graph and robustifies existing detectors with framing and multi-view consistency.
Results
The paper constructs and publicly releases 3D Scene Graph results on the Gibson dataset, with each proposed mechanism improving final accuracy and qualitative detections.
Takeaways & Limitations
Grounding semantics in 3D space supports relationships among objects, space, and cameras while enabling analysis and tasks across semantic information sources.
Takeaways & Limitations
Despite automation, sufficiently accurate results still rely largely on human interaction.
Abstract
from arXiv · showhide
A comprehensive semantic understanding of a scene is important for many applications - but in what space should diverse semantic information (e.g., objects, scene categories, material types, texture, etc.) be grounded and what should be its structure? Aspiring to have one unified structure that hosts diverse types of semantics, we follow the Scene Graph paradigm in 3D, generating a 3D Scene Graph. Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e.g., class, material, and other attributes), rooms (e.g., scene category, volume, etc.) and cameras (e.g., location, etc.), as well as the relationships among these entities. However, this process is prohibitively labor heavy if done manually. To alleviate this we devise a semi-automatic framework that employs existing detection methods and enhances them using two main constraints: I. framing of query images sampled on panoramas to maximize the performance of 2D detectors, and II. multi-view consistency enforcement across 2D detections that originate in different camera locations.
1. Introduction
The paper grounds diverse semantic information in 3D space and organizes it as a layered scene graph connecting building, room, object, and camera entities. It proposes a mainly automatic pipeline using framing and multi-view consistency to make semantic recognition more robust and reduce manual labor.
- Motivation: 3D space provides a more stable grounding for semantics while remaining connected to image, video, and depth outputs.Grounding information on a building’s 3D mesh supports free 3D, amodal, occlusion, and open-space analysis.
- Representation: A 3D Scene Graph extends scene graphs with essential 3D information to represent diverse semantics and relationships.Unlike representations limited to object classes or captions, it combines structure, attributes, and 3D grounding.
- Representation: The graph uses layers for buildings, rooms, objects, and cameras, with attributes on nodes and relationships represented by edges.The camera layer associates camera locations with possible observations such as RGB images.
- Pipeline: The construction pipeline combines existing recognition algorithms with framing and multi-view consistency to automate semantic recognition.These constraints operate during aggregation from 2D observations toward a unified 3D representation.
- Contribution: The paper augments the Gibson Environment database with a publicly available 3D Scene Graph modality.This extends the database with a structured representation of multimodal 3D semantic information.
2. Related Work
Prior work provides structured scene graphs, detectors, and partially automated annotation tools, but existing resources remain fragmented and often require substantial human interaction. The paper combines these foundations toward automated, multimodal 3D semantic organization.
- Scene Graphs: Visual Genome encodes object classes, locations, attributes, and relationships in 2D scene graphs derived from manually defined captions.Its annotations are canonicalized to reduce naming inconsistencies before conversion into the final graph.
- Scene Graphs: Scene graphs have supported generation, captioning, retrieval, visual question answering, relationship detection, and action detection.The paper positions 3D Scene Graphs as extending this scope across 3D space, images, video, and language.
- Structured Prediction: Structured prediction methods such as CRFs and semantic graphs use relationships to promote globally consistent results or facilitate generalization across spaces.Related examples place entities in layers or connect objects, attributes, and affordances for multimodal reasoning.
- Semantic Repositories: Existing object-semantic methods mainly provide class recognition or spatial spans, including segmentation masks and bounding boxes.Examples include Mask R-CNN, BlitzNet, and Frustum PointNet across RGB and RGB-D inputs.
- Annotation: Machine-assisted annotation tools reduce some interaction but remain focused on 2D categories and masks, while 3D systems still rely largely on human labeling.The paper identifies substantial human interaction as a limitation of these approaches.
3. 3D Scene Graph Structure
The 3D Scene Graph represents a scanned space as a four-layered graph spanning buildings, rooms, objects, and cameras, with attributes and relationships connecting its elements.
- The method takes 3D meshes, registered RGB panoramas, and corresponding camera parameters as scanner-style input.Examples of source databases include Matterport3D and Gibson.
- The graph contains building, room, object, and camera layers, with each layer represented by nodes in 3D.The building layer contains the mesh root, rooms compose the second layer, objects the third, and camera locations the fourth.
- Building, room, object, and camera nodes carry attributes such as object class, material type, and pose information.
- Edges connect elements within or across layers to represent relationships such as object-object and camera-object-room connections.
4. Constructing the 3D Scene Graph
The construction pipeline uses existing detectors to automate object annotation, then improves labels through panorama framing, multi-view aggregation, and user verification before deriving further graph elements.
- Annotating RGB and mesh data with object labels and segmentation masks is identified as the main labor bottleneck.
- The pipeline uses framing and multi-view consistency to robustify imperfect 2D detections in image and 3D domains.The experiments use Mask R-CNN detections with confidence scores of 0.7 or higher.
- Framing on Panoramic Images: Framing densely samples rectilinear views over panorama yaw, pitch, and field of view so objects can appear fully captured and nearer image centers.The sampling produces 225 images of size 800 by 800 pixels per panorama.
- Framing on Panoramic Images: Panorama labels are aggregated with weights based on detector confidence and distance from the rectilinear image center, followed by candidate voting and connected components.
- Multi-view consistency: Projected labels from multiple panoramas are combined on the mesh using weighted voting, allowing different viewpoints to correct projection and registration errors.The multi-view weights reflect the heuristic that closer cameras provide larger and better-visible observations.
- User-in-the-loop verification: Manual verification lets users check labels and masks, add missing masks, and correct automatically extracted results through web interfaces.The verification was crowd-sourced through Amazon Mechanical Turk, while unverified automated results were reported as sufficiently robust for certain practical uses.
- Room elements and additional attributes are computed with off-the-shelf or analytical methods, while material attributes are defined manually because existing networks lacked adequate accuracy.
5. Experiments
The experiments evaluate the mainly automatic pipeline on 2D panoramas and 3D meshes, then assess its scene-graph prediction applications. Framing and multi-view consistency improve detections and support more consistent 3D results.
- Automated pipeline evaluation: The pipeline is evaluated on Gibson buildings using 2D panoramas, 3D meshes, and COCO-style average precision and recall.The database provides 572 real indoor buildings with meshes, RGB panoramas, and camera poses; object, material, and texture categories come from COCO, MINC, and DTD.
- Detection results: Each mechanism adds an accuracy boost, progressively removing erroneous detections in qualitative panorama and 3D results.Framing removes tree detections and recovers a highly occluded toilet; adding multi-view consistency removes a painted vase and bed reflection, producing results close to ground truth.
- Using different detectors: The robustification mechanisms provide similar relative AP increases for Mask R-CNN and the faster but less accurate BlitzNet detector.Mask R-CNN has reported COCO AP 41.5, while BlitzNet has reported COCO AP 34.1; the experiments report similar relative increases across baselines.
- 2D scene graph prediction: The generated scene-graph output supports relationship classification for spatial order and relative volume, plus amodal mask segmentation.Amodal segmentation predicts visible and occluded regions from a camera location, using 3D semantic information to infer full occlusion extents without additional annotations.
6. Conclusion
The paper presents a unified 3D Scene Graph for multimodal semantics and relationships, constructed mainly automatically with framing and multi-view consistency. On Gibson, the approach demonstrates robustness improvements and publicly releases the resulting scene graphs.
- Conclusion: The 3D Scene Graph unifies semantic information and relationships among objects, 3D space, and cameras.The conclusion describes a mainly automatic construction approach using framing and multi-view consistency, demonstrated on the Gibson dataset.