Source-linked AI summary
Multi-view PointNet for 3D Scene Understanding
Maximilian Jaritz, Jiayuan Gu, Hao Su
TL;DR
3D perception needs to combine dense image information with sparse point clouds, but the modalities have incompatible spatial structures. MVPNet computes multi-view image features in 2D, lifts and aggregates them into 3D, and fuses them with point-based processing; experiments report stronger point-cloud results, robustness to sparse clouds, and useful design insights across tasks.
Problem
2D–3D fusion is difficult because the modalities have different spaces, correspondences, neighborhood definitions, and resolutions, while point-cloud geometry alone can miss distinctions between flat-surfaced objects.
Method
MVPNet computes features from selected multi-view images, lifts them into 3D, adaptively aggregates them into the input point cloud, and fuses them in canonical 3D space with a point-based network.
Results
MVPNet outperforms published point-cloud networks, shows robustness to very sparse point clouds, and demonstrates benefits from dense image features through extensive experiments.
Takeaways & Limitations
Early feature fusion of dense multi-view images with sparse 3D points provides a practical framework for 3D semantic segmentation and can extend to 3D instance segmentation.
Takeaways & Limitations
With sliding-window processing, MVPNet can outperform state-of-the-art point-based approaches, but whole-scene methods such as SCN have a clear advantage.
Abstract
from arXiv · showhide
Fusion of 2D images and 3D point clouds is important because information from dense images can enhance sparse point clouds. However, fusion is challenging because 2D and 3D data live in different spaces. In this work, we propose MVPNet (Multi-View PointNet), where we aggregate 2D multi-view image features into 3D point clouds, and then use a point based network to fuse the features in 3D canonical space to predict 3D semantic labels. To this end, we introduce view selection along with a 2D-3D feature aggregation module. Extensive experiments show the benefit of leveraging features from dense images and reveal superior robustness to varying point cloud density compared to 3D-only methods. On the ScanNetV2 benchmark, our MVPNet significantly outperforms prior point cloud based approaches on the task of 3D Semantic Segmentation. It is much faster to train than the large networks of the sparse voxel approach. We provide solid ablation studies to ease the future design of 2D-3D fusion methods and their extension to other tasks, as we showcase for 3D instance segmentation.
1. Introduction
MVPNet addresses difficult 2D–3D fusion by lifting dense multi-view image features into sparse point clouds and combining them in canonical 3D space. The approach improves point-cloud scene understanding, especially under sparse sampling and for objects with weak geometric cues.
- Motivation: 2D–3D fusion is challenging because the modalities have different mappings, neighborhood definitions, spatial resolutions, and sampling characteristics.A Velodyne HDL-64 point cloud projected into an image covers only 5.9% of pixels.
- Motivation: Point-cloud geometry alone struggles to distinguish flat-surfaced objects, while naively adding XYZRGB provides only marginal improvement over XYZ.Examples include doors, refrigerators, and curtains.
- Method: MVPNet computes 2D features from heuristically selected multi-view frames, lifts them into 3D, adaptively aggregates them into the original point cloud, and applies PointNet++ for segmentation.The resulting representation combines image and geometry features in canonical 3D space.
- Method: Lifting image features preserves contextual information from the 2D network’s receptive field while enabling complementary RGB and geometry features to be jointly processed in 3D.The framework is designed to be added to different 3D networks.
- Results: MVPNet outperforms published point-cloud methods by using dense image information while handling occlusions.The paper also reports extensive ablations on fusion design choices and robustness to very sparse point clouds.
2. Related Work
Prior 3D scene-understanding methods use voxel CNNs or point-cloud networks, while 2D–3D lifting methods transfer image features through pixel-level correspondences or higher-level proposals. MVPNet builds on point-cloud processing while targeting feature fusion in continuous 3D space.
- 2D to 3D Lifting: 2D–3D lifting methods transfer image features into voxel volumes, bird’s-eye-view maps, or 3D frustums using pixel-level correspondences or object proposals.These approaches differ in whether they lift low-level feature maps or high-level 2D detections.
- 2D to 3D Lifting: MVPNet’s lifting uses continuous 3D distances without discretization errors.This design distinguishes its point-cloud formulation from grid-based voxel processing.
- 3D Networks: Voxel-based 3D CNNs discretize point clouds into grids, whereas point-cloud networks operate directly on points and preserve inherent sparsity.MVPNet uses point-cloud networks for this sparsity advantage.
- 3D Semantic Segmentation: PointNet uses shared MLPs and global max-pooling, while PointNet++ adds hierarchical set-abstraction layers better suited to whole-scene analysis.The hierarchical structure scales more effectively to larger point sets.
- 3D Instance Segmentation: The paper frames 3D instance segmentation as predicting exact point masks rather than bounding boxes and identifies proposal-based, point-proposal, and clustering-based strategies.MVPNet is later extended to this task.
3. MVPNet
MVPNet selects informative RGB-D views, lifts their 2D features into a dense 3D point cloud, aggregates them into the sparse input points, and fuses them with geometry using PointNet++.
- View Selection: MVPNet selects 1 to 5 RGB-D views per scene chunk to maximize coverage while avoiding redundant processing of overlapping video frames.Training uses a greedy strategy that selects the frame covering the most previously uncovered points.
- 2D-3D Feature Lifting: A 2D U-Net encoder-decoder computes feature maps from selected RGB images, which are unprojected using depth, camera intrinsics, and poses into a dense 3D feature cloud.For M feature maps, concatenation produces S_dense with MNRGB feature points.
- 2D-3D Feature Aggregation: A shared-MLP aggregation module distills features from k nearest dense-cloud neighbors into each sparse input point using image features and point-distance features.The resulting multi-view feature-augmented point cloud associates the distilled features with the original 3D coordinates.
- 2D-3D Feature Aggregation: The lifting module is differentiable, enabling end-to-end MVPNet training after 2D features are transferred to the sparse point cloud.The augmented point cloud is the representation passed to the 3D fusion network.
- 3D Fusion Network: PointNet++ hierarchically encodes and decodes the augmented point cloud for semantic prediction, with input geometry and image features fused through early concatenation.Intermediate and late fusion strategies were also investigated, but early fusion is the adopted architecture.
4. Experiments
Experiments on ScanNetV2 show that MVPNet improves point-cloud semantic segmentation, remains robust as point density decreases, and trains faster than heavyweight sparse-voxel methods.
- 3D Semantic Segmentation: MVPNet outperforms published point-cloud methods on the ScanNetV2 3D semantic-label benchmark, especially for classes with weak geometric cues.The reported gains are attributed to lifting image features into 3D, including for refrigerators, pictures, and curtains.
- Voxel-Based Comparison: Against voxel-based methods, 3DMV does not match MVPNet's performance or inference time, requiring 500 s/scene versus 3.35 s/scene.3DMV is described as a similar joint 2D-3D network in the voxel-based domain.
- Robustness to Varying Point Cloud Density: MVPNet does not match SCN's result, but is more robust to low-resolution point clouds and is relevant to sensors producing sparse data.The paper motivates this robustness by noting that real-world point clouds are often much sparser than images.
- Runtime: MVPNet converges in 20 hours on a GTX 1080Ti, whereas heavyweight SCN takes 12 days on the same GPU or 4 days on a V100.The comparison includes 2D encoder-decoder pretraining for MVPNet.
- 3D Instance Segmentation: The method is also extended to 3D instance segmentation, where MVPNet features improve validation mAP from 38.8 to 47.1.The extension modifies R-PointNet to consume semantic features from MVPNet.
5. Ablation Studies
Ablations show that view coverage, early fusion, complementary geometry, and stronger backbones shape MVPNet performance, while aggregation choices matter most under limited coverage.
- Number of Views: Three views exceed 90% coverage, reaching 92.9% in the reported setting; additional views improve mIoU but increase computation.One view reaches 68.1% coverage, and three frames are selected as the default trade-off.
- Feature Aggregation Module: With one view, using three nearest neighbors instead of one improves performance by at least 0.8 mIoU.The paper attributes this to limited single-view coverage and possible feature-consistency benefits from multiple neighbors.
- Feature Aggregation Module: With three views, nearest-neighbor count does not affect performance; the final configuration uses three neighbors, an MLP, and sum aggregation.Coverage is already high, while the MLP can slightly improve consistency between 2D and 3D representations.
- Fusion Strategies: Removing XYZ coordinates from early fusion reduces performance by -2.2 mIoU, showing that geometry remains complementary to image features.The result holds despite PointNet++ already inducing a geometric hierarchy.
- Backbones: Stronger 2D backbones raise mIoU, with ResNet34 performing best; doubling sampled 3D centroids adds 1.4 mIoU, while MSG adds none.VGG16 remains the ablation backbone for runtime reasons, while ResNet34 is used for best performance.
6. Conclusion
MVPNet fuses multi-view image and point-cloud features in canonical 3D space, while whole-scene sparse voxel methods retain an advantage over its sliding-window processing.
- MVPNet outperforms state-of-the-art point-based approaches by a significant margin when using sliding-window processing.
- Whole-scene methods such as SCN have a clear advantage over MVPNet's sliding-window approach.
- The framework computes multi-view image features in 2D, lifts them to 3D, and fuses them with geometry in canonical 3D space.
- Qualitatively, MVPNet avoids some PointNet++ errors on similarly shaped objects, including doors, refrigerators, and curtains.
- MVPNet can misclassify a bookshelf as a door when similar wooden appearance and spatial proximity create ambiguity.
A. 2D Encoder Decoder Architecture
The 2D encoder-decoder architecture uses a VGG16 encoder and a decoder that combines skip-connected features with upsampling.
- The 2D encoder-decoder network is inspired by U-Net and uses VGG16 initialized with ImageNet pre-trained weights.
- The decoder fuses concatenated skip-connection features with convolution and upsamples using transposed convolution.
B. Comparison with SparseConvNet (SCN)
The comparison references lightweight SCN, a small U-Net using 5cm-cubed voxels, with implementation details available in released code.
- Lightweight SCN is described as a small U-Net operating on 5cm-cubed voxels.
C. Experiments on S3DIS
On S3DIS, MVPNet uses images and xyx-maps from 2D-3D Semantics and improves over previous methods by 4.16 mIoU.
- 4.16 mIoU improvement over previous methods is reported for MVPNet on S3DIS.
- The evaluation uses images and xyx-maps from the 2D-3D Semantics dataset.
D. More Ablation Studies
Ablations on ScanNetV2 evaluate implementation variants and show that backbone choice, class weighting, and ensembling affect MVPNet performance.
- The ScanNetV2 validation variants use MVPNet with five views and a ResNet34 2D backbone.
- The reported point-cloud network uses centroid counts of 2048, 512, 128, and 64.
- A stronger ResNet34 backbone improves mIoU by 0.7 over VGG19, while class weighting adds another 0.7 mIoU.
- Ensembling four MVPNet models with ResNet34 achieves the best reported performance of 68.3.