Source-linked AI summary
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
Angela Dai, Matthias Nießner
TL;DR
3DMV addresses 3D semantic segmentation of indoor RGB-D scans, where existing approaches use either geometry or RGB information. It jointly processes 2D RGB features and 3D geometry, backprojects and pools multi-view features, and reports 75% accuracy versus 52.8% for the best existing volumetric architecture.
Problem
3D semantic segmentation needs dense spatial labels from RGB-D reconstructions, but existing approaches do not jointly leverage RGB and geometric information.
Method
3DMV uses a joint end-to-end 2D-3D network that backprojects learned RGB image features into a volumetric grid and combines them with 3D geometric features.
Results
75% average classification accuracy is achieved, improving 22.2% over existing volumetric methods and 14.8% over PointNet++.
Takeaways & Limitations
The evaluations show that 2D and 3D features complement each other and that multiple RGB views can improve performance through higher coverage.
Takeaways & Limitations
Dense volumetric grids become impractical at high resolutions, and independently predicted scene columns can produce label inconsistencies.
Abstract
from arXiv · showhide
We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. Rather than simply projecting color data into a volumetric grid and operating solely in 3D -- which would result in insufficient detail -- we first extract feature maps from associated RGB images. These features are then mapped into the volumetric feature grid of a 3D network using a differentiable backprojection layer. Since our target is 3D scanning scenarios with possibly many frames, we use a multi-view pooling approach in order to handle a varying number of RGB input views. This learned combination of RGB and geometric features with our joint 2D-3D architecture achieves significantly better results than existing baselines. For instance, our final result on the ScanNet 3D segmentation benchmark increases from 52.8\% to 75\% accuracy compared to existing volumetric architectures.
1 Introduction
3D semantic segmentation extends image labeling with spatial scene understanding, which is especially relevant to robotics. 3DMV addresses RGB integration by jointly learning from 2D RGB features and 3D geometry in an end-to-end network.
- 3D segmentation predicts per-voxel semantics while preserving a spatial map of scene content.This spatial semantic representation is particularly relevant to robotics, where systems need to know both what objects are and where they are.
- Existing volumetric approaches can incorporate RGB by adding color channels, but 3DMV instead learns a joint RGB-geometry representation.The method targets the problem of incorporating RGB information into 3D semantic segmentation without relying only on voxelized color.
- 3DMV extracts full-resolution 2D image features, backprojects them into 3D, and processes them alongside learned geometric features.The resulting architecture combines 2D feature extraction with 3D convolutional processing in an end-to-end approach.
- 75% accuracy is achieved, improving from 52.8% over the best existing volumetric architecture.The reported comparison is for 3D segmentation accuracy.
2 Related Work
Prior work established volumetric 3D CNNs, multi-view feature aggregation, and 3D semantic segmentation benchmarks. 3DMV draws on multi-view representations while retaining geometry and targeting dense per-voxel semantic prediction.
- 3D CNNs extend convolutional processing to volumetric domains but face substantial memory and compute requirements.Hierarchical 3D CNNs were proposed to represent and process volumes more efficiently.
- Multi-view CNNs aggregate features from rendered views, but object-classification outputs do not spatially correlate those features.Later mesh-segmentation methods projected 2D confidence maps onto geometry, but handled relatively small label sets and used renderings as replacement input for 3D geometry.
- Other multi-view methods use recurrent or unsupervised feature accumulation for stereo reconstruction and latent 3D-space learning.A separate alternative projects colors directly into voxels, representing another way to combine views with 3D input.
- 3D semantic segmentation predicts semantic labels per voxel and is especially relevant to robotics because spatial semantic understanding is essential.Datasets and benchmarks were developed for this relatively recent task.
- ScanNet introduced a benchmark built from approximately 1.5k RGB-D scans and reconstructions, with annotations on the 3D reconstructions.It also includes approximately 2.5 million RGB-D frames with 2D annotations derived from rendered 3D-to-2D projections.
3 Overview
3DMV predicts dense per-voxel semantic labels from commodity RGB-D scans by jointly using a volumetric scene representation and nearby RGB views.
- The method uses a regular volumetric grid whose voxels encode known-occupied, known-free, or unknown geometry.The network predicts semantic class labels for each voxel of the reconstructed scene.
- 3DMV operates on scene chunks and predicts labels for center columns while sliding the subvolume through the reconstruction at test time.This chunk-based formulation supports semantic segmentation of full 3D scenes.
- The overview introduces the network architecture before describing training and implementation.The architecture is presented in Section 4, followed by training and implementation in Section 5.
4 Network Architecture
The network combines shared 2D RGB streams with a 3D geometry stream. RGB features are backprojected and pooled across views, fused with geometric features, and used for per-voxel prediction on scene chunks.
- 4 Network Architecture: The architecture combines a 3D geometry stream with several shared-weight 2D RGB streams.The 3D stream processes volumetric scan geometry, while the 2D streams process associated RGB images.
- 4.2–4.4 RGB and Joint Processing: Multiple aligned RGB views are feature-extracted in 2D, backprojected differentiably into 3D, max-pooled per voxel, and processed with 3D convolutions alongside geometry.The two 3D streams are joined before predicting per-voxel labels, and the complete network is trained end to end.
- 4 Network Architecture: Aligned RGB-D sequences are fused into a volumetric grid, with known camera poses connecting images to the reconstruction.The method assumes RGB-D images are aligned in world coordinates and supports sensor types beyond the examples given.
- 4.1 3D Network: The 3D network processes 31×31×62 voxel subvolumes at 4.8cm resolution and predicts 62 labels for each center column.The subvolume covers a 1.5m × 1.5m spatial neighborhood and 3m in height.
- 4.3 Backprojection: Depth data prunes projected voxels beyond a 4.8cm threshold, restricting associations to voxels near the depth-map geometry.Voxel-to-pixel correspondences are computed on the fly using known 6-DoF alignments.
- 4.4 Joint 2D-3D Network: The joint network concatenates independently processed RGB and geometric features, then applies further 3D convolutions before classification.The authors tested several fusion locations within the 3D network.
- 4.4 Joint 2D-3D Network: Most learnable parameters belong to the combined 3D stream immediately before per-voxel prediction.This parameter distribution is associated with reliance on strong feature maps at that stage.
- 4.5 Full-Scene Inference: Large scenes are segmented by sliding fixed-height subvolumes across the reconstruction’s xy-domain.The 3m subvolume height is intended to cover most indoor environments.
5 Training
3DMV trains its joint 2D-3D network end-to-end on correlated volumetric and RGB inputs, using selected nearby views and GPU-implemented backprojection.
- 5 Training: The network receives correlated 3D geometry and RGB views for end-to-end training.Geometry uses a ternary occupancy grid split into occupancy and known-versus-unknown channels.
- 5 Training: Nearby RGB images are greedily selected by the number of currently uncovered voxels in each subvolume.The method typically selects 3–5 images because additional coverage gains diminish.
- 5 Training: The custom backprojection layer is implemented as parallelized matrix multiplications and runs on the GPU through PyTorch.End-to-end optimization performed best compared with training only parts of the network.
- 5 Training: Training uses SGD with a learning rate of 0.001, momentum 0.9, and batch size 8.The reported optimizer settings are used for the training process.
6 Results
On ScanNet, 3DMV combines RGB and geometry more effectively than single-modality or volumetric baselines, with accuracy improving as additional views increase coverage, though gains diminish.
- Comparison to state of the art: 75% average classification accuracy is achieved by the best 5-view, end-to-end 3DMV model on ScanNet.This improves 22.2% over existing volumetric approaches and 14.8% over PointNet++.
- RGB and geometric inputs: 54.4% to 70.1% accuracy results from adding RGB features to the geometry-only network, whereas per-voxel colors reach only 55.9%.The authors attribute the marginal per-voxel-color gain to limited grid resolution of approximately 5 cm.
- RGB and geometric inputs: For 3 views, the joint network reaches 73.0% accuracy, exceeding geometry-only performance and RGB-label backprojection with 3D convolutions.The reported synergies improve over the individual inputs by 14.8% and 18.6%, respectively.
- Feature fusion: Fusion after the second third of the 3D network performs best among tested combinations, whose accuracies range from 65.4% to 69.1%.The second-third option is used as the default in other experiments.
- Additional views: 54.4% geometry-only accuracy rises to 70.1% with 1 view, 73.1% with 3 views, and 75.0% with 5 views.The incremental gains are +15.7%, +3.0%, and +1.9%, respectively, and decrease as views accumulate.
- Training choices: End-to-end training outperforms a fixed 2D network by 1.0%, 0.2%, and 0.5% for 1, 3, and 5 views.The end-to-end variants are consistently better, although the improvements are smaller than initially hoped.
- Additional views: RGB-view coverage increases from 40.3% with 1 view to 64.4% with 3 views and 72.3% with 5 views.Qualitative results compare 3DMV with ScanNet, ScanComplete, and PointNet++ on the ScanNet test set.
- Limitations: Dense volumetric grids become impractical at high resolutions, and sparse approaches such as OctNet are suggested as a possible remedy.The limitation is especially relevant because RGB-D reconstructions can use sub-centimeter voxel resolutions.
7 Conclusion and Future Work
3DMV combines geometric and RGB features in a joint 3D-multi-view architecture for semantic 3D scene segmentation. The authors report higher accuracy than prior 3D approaches while identifying several open directions for future work.
- 3DMV combines geometric and RGB features in a joint network architecture for semantic 3D scene segmentation.
- More than 14% higher classification accuracy is reported than with the best existing 3D segmentation approach.
- Future work includes 3D semantic and instance segmentation, scene representations for mixed sparse-dense data, and multimodal generative reconstruction tasks.
A Evaluation on Matterport3D [33]
The paper evaluates 3DMV on building-scale Matterport3D RGB-D reconstructions against volumetric semantic 3D segmentation methods. Results are also examined across one, three, and five input views.
- Matterport3D contains 90 building-scale RGB-D reconstructions with dense annotations similar to ScanNet.
- The final model uses five views and is compared with ScanNet and ScanComplete on Matterport3D.
- Each additional input view improves results by several percentage points in the one-, three-, and five-view evaluation.
B Additional Qualitative Results
Additional qualitative results are presented for ScanNet and Matterport3D test sets. The Matterport3D examples compare 3DMV with existing volumetric semantic segmentation approaches.
- Additional qualitative 3D semantic segmentation results are shown on the ScanNet test set.
- Black regions denote unannotated areas or labels outside the evaluated label set.
- Matterport3D qualitative results compare 3DMV with the ScanNet and ScanComplete 3D-based approaches.