Source-linked AI summary
PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji
TL;DR
Existing semantic mapping systems lack holistic 3D understanding that combines dense stuff labeling with individual thing discrimination. PanopticFusion addresses this with online volumetric panoptic mapping, and on ScanNet v2 it outperformed or compared with state-of-the-art offline 3D DNN methods in semantic and instance segmentation.
Problem
Semantic mapping needs dense volumetric reconstruction together with dense semantics and individual object discrimination for robotics and augmented-reality applications.
Method
PanopticFusion fuses 2D semantic and instance predictions, tracks and integrates panoptic labels with depth in a spatially hashed volumetric map, and applies online CRF regularization.
Results
PanopticFusion outperformed or compared with state-of-the-art offline 3D DNN methods in both 3D semantic and instance segmentation tasks on ScanNet v2.
Takeaways & Limitations
The system provides online stuff-and-things scene understanding with large-scale reconstruction, labeled mesh extraction, and a demonstrated augmented-reality application.
Abstract
from arXiv · showhide
We propose PanopticFusion, a novel online volumetric semantic mapping system at the level of stuff and things. In contrast to previous semantic mapping systems, PanopticFusion is able to densely predict class labels of a background region (stuff) and individually segment arbitrary foreground objects (things). In addition, our system has the capability to reconstruct a large-scale scene and extract a labeled mesh thanks to its use of a spatially hashed volumetric map representation. Our system first predicts pixel-wise panoptic labels (class labels for stuff regions and instance IDs for thing regions) for incoming RGB frames by fusing 2D semantic and instance segmentation outputs. The predicted panoptic labels are integrated into the volumetric map together with depth measurements while keeping the consistency of the instance IDs, which could vary frame to frame, by referring to the 3D map at that moment. In addition, we construct a fully connected conditional random field (CRF) model with respect to panoptic labels for map regularization. For online CRF inference, we propose a novel unary potential approximation and a map division strategy. We evaluated the performance of our system on the ScanNet (v2) dataset. PanopticFusion outperformed or compared with state-of-the-art offline 3D DNN methods in both semantic and instance segmentation benchmarks. Also, we demonstrate a promising augmented reality application using a 3D panoptic map generated by the proposed system.
I. INTRODUCTION
PanopticFusion targets holistic online 3D scene understanding by combining dense stuff labeling with individual thing discrimination, while supporting large-scale volumetric reconstruction. It integrates 2D panoptic predictions into a regularized volumetric map and achieves competitive performance on ScanNet v2.
- 3D semantic mapping matters for robotics and context-aware augmented reality because it combines geometric reconstruction with recognition of the environment.
- An ideal system should densely reconstruct large-scale scenes volumetrically and provide both dense semantic labels and individual object discrimination.Dense semantics support robot navigation, while individual-object discrimination supports robot–object interaction.
- PanopticFusion is the first reported semantic mapping system to understand scenes at the level of stuff and things.It predicts class labels for stuff regions and class labels plus instance IDs for thing regions at the pixel level.
- The system fuses 2D semantic and instance segmentation outputs, tracks instance IDs through the volumetric map, integrates labels with depth, and regularizes the map using a fully connected CRF.Its CRF inference uses a unary-potential approximation and map division strategy; the latter reduces computation without an accuracy drop.
- On ScanNet v2, PanopticFusion achieved superior or comparable performance to state-of-the-art offline 3D DNN methods in 3D semantic and instance segmentation.The system also demonstrated an augmented-reality application using its generated 3D panoptic map.
- Spatially hashed volumetric mapping enables large-scale 3D reconstruction and labeled mesh extraction.
II. RELATED WORK
Prior semantic mapping systems largely separated dense scene labeling from object-oriented recognition, whereas PanopticFusion combines dense stuff labeling, individual object discrimination, and scalable online volumetric reconstruction.
- Semantic mapping systems are divided into dense labeling and object-oriented approaches based on their semantic representations.
- Dense labeling approach: Dense labeling methods assign class labels or class probabilities to surfels or voxels to produce dense 3D semantic segmentation.
- Object-oriented approach: Object-oriented methods use 3D model databases or geometric and descriptor matching, but database-based approaches require exact shape correspondence.
- PanopticFusion stores DNN-predicted panoptic labels in a single volumetric map, allowing dense semantic labeling and discrimination of individual objects.
- Voxel hashing manages spatially hashed voxel blocks, improving memory efficiency over a single voxel grid and enabling large-scale reconstruction.
C. 2D Panoptic Label Prediction
PanopticFusion fuses 2D semantic and instance segmentation outputs into pixel-wise panoptic labels, then resolves frame-to-frame instance-ID inconsistency by referencing the volumetric map.
- PSPNet predicts pixel-wise class labels while Mask R-CNN predicts per-pixel instance IDs, which are fused into panoptic labels.
- Directly integrating raw panoptic labels causes inconsistency because Mask R-CNN may assign different instance IDs to the same object across frames.
- The system uses map-based reference labels rather than a visual tracker, supporting object re-identification during loopy camera trajectories.
- Instance IDs are associated using intersection over union, with the best match accepted above threshold θU and a new ID assigned otherwise.
- Associations are processed in descending mask-area order and use exclusive labels to avoid foreground-object under-segmentation.
E. Volumetric Integration
The system integrates depth-derived geometry and color along sensor rays while maintaining discrete panoptic labels with an increment/decrement weight update instead of averaging them.
- For each pixel, the system casts a ray to a back-projected 3D point and updates voxels along the ray within a truncated distance.
- TSDF values are updated by weighted averaging using a quadric weight that reflects depth-measurement reliability, and voxel colors receive similar updates.
- Panoptic labels are stored as a single discrete label per voxel because weighted averaging is unsuitable and recording all labels increases memory use.
- When the estimated and stored panoptic labels coincide, the voxel label weight is increased; when they differ, the weight is decreased.
- If the new label's weight exceeds the previous voxel-label weight, the voxel label is replaced with the newly estimated label.
F. Thing Label Probability Integration
Because thing-class predictions can remain uncertain even with accurate masks, PanopticFusion integrates thing labels probabilistically and weights distributions by detection confidence.
- Thing labels are probabilistically integrated because Mask R-CNN predictions can be uncertain when only a small part of an object is visible.
- The integrated distribution combines the foreground-object probability with the conditional distribution over thing labels.
- Weighting by detection confidence makes the final distribution preferentially reflect reliable detections.
G. Online Map Regularization
PanopticFusion further improves recognition accuracy by regularizing its reliable 3D panoptic map with a fully connected CRF.
- A fully connected CRF with Gaussian edge potentials regularizes the 3D panoptic map to further improve recognition accuracy.The approach follows efficient inference methods previously used for 2D image segmentation and related 3D maps.
1) Problem Setting:
The system formulates map regularization as panoptic-label inference over voxel nodes, using approximate unary potentials and Gaussian pairwise terms while leveraging stored labels and weights.
- The CRF assigns a panoptic label variable to each voxel and minimizes Gibbs energy using mean-field approximation and message passing.
- Unary potentials use the negative logarithm of a panoptic-label probability distribution, following standard class-label CRF practice.
- Pairwise potentials combine Gaussian kernels that regularize voxel labels according to voxel colors and spatial locations.The label disagreement term is a simple Potts model.
- Because each voxel stores only one label rather than a full probability distribution, unary potentials are approximated from that label and its stored weights.The approximation is motivated by memory efficiency and relies on an assumption described in the method.
- The approximation cannot exactly calculate probabilities for alternative labels because previously integrated labels are not fully recorded, so those probabilities are estimated.Despite this approximation, the resulting unary potential improves recognition accuracy quantitatively and qualitatively.
3) Map Division for Online Inference:
To make online fully connected CRF inference practical, PanopticFusion divides the volumetric map into spatially contiguous submaps, reducing the effective label count per inference problem.
- CRF inference has complexity O(NM), but the potentially large number of panoptic labels can make online inference impracticable.The paper notes that the label count may reach several hundreds in practice.
- Dividing the volumetric map into S spatially contiguous submaps is proposed to address the online-inference cost.
IV. EVALUATION
PanopticFusion was evaluated on ScanNet v2 using 2D recognition modules, quantitative segmentation benchmarks, and qualitative map and regularization experiments. It achieved comparable or superior performance to offline 3D DNN methods while retaining an online incremental framework, though some 3D methods remained stronger on certain semantic measures.
- Dataset and setup: ScanNet v2 provides RGB-D images, camera trajectories, reconstructed 3D models, and semantic annotations across 1,201 training and 312 open test scenes.The experiments used 640×480 RGB-D images and provided camera trajectories for fair comparison.
- Dataset and setup: The system used PSPNet with a ResNet-50 backbone and Mask R-CNN with a ResNet-101-FPN backbone for 2D semantic and instance recognition.
- Dataset and setup: The integration used 0.024 m voxels, 16×16×16-voxel blocks, an IoU threshold of θU = 0.25, and five CRF iterations.
- Quantitative and qualitative results: PanopticFusion achieved comparable or superior performance to state-of-the-art offline 3D DNN methods on 3D semantic and instance segmentation.Its classwise advantages were especially notable for small objects and objects difficult to recognize from geometry alone.
- Quantitative and qualitative results: Figure 3 presents qualitative 3D panoptic maps from three ScanNet v2 scenes, with ground truth and predictions using different reconstruction algorithms.Thing colors in the predictions are not necessarily the same as in the ground truth.
- Quantitative and qualitative results: Figure 4 relates the maximum number of voxel blocks to recognition accuracy and logarithmically scaled computational time.
- Quantitative and qualitative results: Several semantic segmentation methods outperformed PanopticFusion because of their large 3D receptive fields, while PanopticFusion operated online and incrementally.Applying 3D DNNs to partial observations and integrating them online remained future work.
C. Evaluation of Map Regularization
Map regularization improved recognition while map division substantially reduced computational time without reducing recognition performance. The online system therefore used divided CRF regularization, achieving interactive throughput on a large ScanNet scene.
- Recognition performance improved with the proposed unary potential approximation whether or not map division was used.
- Map division drastically reduced computational time without decreasing recognition performance versus constructing a CRF model for the whole map.
- The online system used map division with a maximum of 25 voxel blocks because it provided better recognition accuracy and acceptable computational time.
- Online map regularization improved recognition performance, boosting scores for almost all classes.
- 4.3 Hz was the system throughput on a typical large-scale ScanNet v2 scene, with Mask R-CNN as the bottleneck process.
V. APPLICATIONS
PanopticFusion's online 3D panoptic meshes support context-aware augmented-reality visualizations, including semantic path planning, object interaction, and natural occlusion and collision effects.
- 3D panoptic meshes enable path planning on stuff regions such as floors and walls.
- The application supports interaction with individual thing regions and interactions appropriate to each region's semantics.
- Natural occlusion and collision visualization are supported by the reconstructed 3D panoptic map.
- In the demonstrated application, humanoids and insect-type robots navigate floor and wall meshes through automatic path planning.
- Semantic interactions include humanoids sitting or lying on chairs and sofas, while computer-generated objects appear on tables.
- The authors identify potential usefulness beyond AR, including autonomous robots that explore scenes and manipulate objects.