Source-linked AI summary
Virtual Multi-view Fusion for 3D Semantic Segmentation
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru
TL;DR
The paper addresses limitations of view-based 3D semantic segmentation for reconstructed scenes by selecting and rendering informative virtual views, then fusing their predictions on mesh surfaces. On ScanNet, this approach outperforms prior multiview methods, is competitive with recent 3D convolution methods, and performs especially well with fewer training scenes or inference views.
Problem
View-based 3D semantic segmentation struggles with occlusion, lighting variation, and camera-pose misalignment, limiting performance on standard benchmarks despite pretrained image-processing networks.
Method
The method renders synthetic images from carefully selected virtual views using wide fields of view and additional channels, trains a 2D segmentation model, and fuses predictions on 3D mesh surfaces.
Results
The method outperforms previous multiview approaches, remains competitive with recent 3D convolution methods, and can outperform all approximately 1,700 original views using approximately 12 virtual views per scene at inference.
Takeaways & Limitations
Careful virtual-view selection makes multiview fusion a viable alternative to 3D convolution for semantic segmentation of textured meshes.
Takeaways & Limitations
The study treats 3D segmentation as an offline task and does not optimize computation cost during training or inference, allowing as many virtual views as needed.
Abstract
from arXiv · showhide
Semantic segmentation of 3D meshes is an important problem for 3D scene understanding. In this paper we revisit the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes. Given a 3D mesh reconstructed from RGBD sensors, our method effectively chooses different virtual views of the 3D mesh and renders multiple 2D channels for training an effective 2D semantic segmentation model. Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels. Using the large scale indoor 3D semantic segmentation benchmark of ScanNet, we show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results compared to all prior multiview approaches and competitive with recent 3D convolution approaches.
1 Introduction
The paper addresses limitations of view-centric and place-centric 3D semantic segmentation by using synthetic virtual views, and reports strong ScanNet performance through multiview fusion.
- 3D semantic segmentation assigns a semantic label to every surface point and supports semantic mapping, site monitoring, and autonomous navigation.
- 3D sparse voxel convolution methods recognize 3D patterns effectively but require substantial memory, limiting spatial resolution and batch sizes.
- RGB-D view-centric methods leverage pretrained 2D image networks but struggle with occlusion, lighting variation, and camera-pose misalignment.
- The method renders synthetic images from virtual views using wide FOVs, strategically selected viewpoints, non-photorealistic rendering, and known camera parameters.
- 3.1% 3D mIoU improvement results from virtual views with original camera parameters over original photographic images; additional channels and higher FOV add 5.7%.
- On ScanNet, the approach outperforms prior multiview methods, remains competitive with recent 3D methods, and can outperform all-view fusion using approximately 12 rather than 1700 views.
2 Related Work
Prior work aggregates predictions from real RGB-D views or processes 3D representations directly, while synthetic rendering offers scale but introduces domain adaptation challenges.
- Multi-view labeling: Earlier multiview methods backprojected 2D semantic predictions onto reconstructed surfaces using weighted averaging, CRFs, Bayesian fusion, or 3D convolutions.
- Multi-view labeling: These methods relied on originally captured photographic views, whose limited fields of view and physical constraints restrict multiview coverage.
- 3D convolution: Recent 3D convolution methods operate on point clouds, meshes, voxel grids, or octrees, but remain limited in spatial resolution compared with 2D images.
- 3D convolution: 3D convolution methods generally train with supervision from much smaller 3D datasets than available 2D image datasets.
- Synthetic data: Synthetic renderings can create image datasets of unlimited size, but models trained on synthetic data generally perform poorly on real data because of domain adaptation.
3 Method Overview
The proposed system selects and renders virtual views during training and inference, applies a 2D segmentation model, and fuses projected predictions in 3D.
- The system overview organizes virtual multiview fusion into training and inference stages.
- Training stage: During training, virtual views, camera parameters, rendered channels, and rendering settings are selected before generating labeled training images for a 2D segmentation model.
- Inference stage: During inference, the trained model segments rendered virtual views, projects semantic features to 3D, and fuses multiple projections to derive 3D categories.
- The proposed view-selection approaches expand the choices of camera extrinsics beyond the original captured views.
4 Virtual view selection
Virtual view selection broadens context and coverage by relaxing physical camera constraints, using multiple channels and scales, and balancing training views while preserving training–inference consistency.
- Virtual view selection permits camera parameters and data augmentations tailored to 2D semantic segmentation, including physically unrealistic but useful viewpoints.
- Camera intrinsics: Higher FOV captures larger scene context than original cameras, improving the contextual information available for 2D semantic segmentation.
- Camera extrinsics: Uniform sampling generates novel top-down and center-looking views from positions distributed across the 3D scene.
- Camera extrinsics: Scale-invariant sampling places cameras at multiple distances from unsupervised scene segments and uses depth checks to reduce foreground occlusion.
- Camera extrinsics: Class-balanced sampling targets mesh segments from under-represented semantic categories and therefore applies only when training labels are available.
- Channels for rendering: Rendering RGB, normals, and normalized global XYZ coordinates supplies view-invariant geometric information beyond RGB-D sensor channels.
- Rendering parameters: Backface culling enables views outside rooms to see more context by preventing wall backfaces from blocking the camera.
- Training vs. inference stage: Training and inference use similar view-selection approaches to avoid a domain gap, while this offline setting does not optimize computation cost and allows as many views as needed.
5 Multiview Fusion
The method projects 2D features from rendered virtual views onto visible 3D points using depth consistency, then averages the valid projected features for fusion.
- The 2D model runs on virtual views to produce image features, such as per-pixel unary probabilities, for 3D fusion.Rendered depth channels support subsequent visibility checks during projection.
- Each 3D point is projected into virtual views, and a feature is accumulated only when the rendered depth matches its point-to-camera distance.This depth-consistency test suppresses features from occluding surfaces.
- The projected feature set F_k contains valid image features from pixels within the valid image domain and below the depth threshold δ.The threshold is defined by |d_i(x_k,i) − c_k,i| < δ.
- The fused feature for each 3D point is the average of all features in F_k, which performs better than selecting the maximum-probability category.The paper compares simple averaging with an alternative maximum-probability fusion rule.
6 Experiments
Experiments evaluate virtual multiview fusion on ScanNet and S3DIS, including benchmark comparisons, component ablations, and view-count studies. The method achieves strong segmentation results and retains good performance with fewer inference views.
- 6.1 Evaluation on ScanNet dataset.: The evaluation compares virtual multiview fusion with 3D point-set and sparse-voxel convolution methods on ScanNet semantic segmentation.The ScanNet experiments cover both 3D mesh and 2D image semantic segmentation, with detailed validation-set ablations.
- 6.1 Evaluation on ScanNet dataset.: 74.6% mean IoU on the ScanNet test set exceeds the previous best multiview result of 52.9%.The paper reports sharper boundaries and more accurate, consistent labels after aggregation on 3D surfaces.
- 6.2 Evaluation on Stanford 3D Indoor Spaces (S3DIS).: 65.4% 3D mIoU on S3DIS exceeds MinkowskiNet at 65.35% and PointASNL at 62.60%.The evaluation follows the Area5 test setting described for the S3DIS benchmark.
- 6.3 Ablation Studies: The ablation study finds significant 3D IoU improvements from virtual renderings, additional normal and coordinate channels, wider field of view, and virtual viewpoint selection.The sequential ablation replaces characteristics of original views with virtual ones.
- 6.3 Ablation Studies: 3.1% 3D mIoU improvement comes from virtual views over photographic images; normal and coordinate channels add 2.9%, and wider field of view adds 1.8%.With fixed camera extrinsics, the paper attributes these gains respectively to removing reconstruction and pose errors, adding 3D information, and increasing context.
- 6.3 Ablation Studies: 68.2% 3D mIoU is reached with approximately 40 virtual views per scene, compared with 70.1% using approximately 2000 views.Performance increases with more views but shows diminishing returns.
7 Conclusion
The paper presents virtual multiview fusion as a strong approach for semantic segmentation of textured 3D meshes. Its design combines virtual views and rendering choices to address limitations of earlier multiview methods.
- The approach combines virtual views, additional channels, back-face culling, wide field of view, and multiscale-aware view sampling.These design choices target 2D–3D misalignment, occlusion, narrow views, and scale invariance.
- Multiview fusion is presented as a viable alternative to 3D convolution for semantic segmentation of textured meshes.The conclusion states that carefully selected and rendered virtual views enable multiview fusion to outperform almost all recent 3D convolution networks.
8 Appendix
The appendix provides qualitative ScanNet validation results and per-class ScanNet test results for 3D and 2D semantic labeling.
- Figures 10 and 11 show qualitative 3D semantic segmentation results for ScanNet validation scenes.The figures cover multiple named validation scenes.
- Tables 4 and 5 provide detailed per-class segmentation IoU scores on the ScanNet test split.Table 4 covers 3D semantic labeling and Table 5 covers 2D semantic labeling.