Source-linked AI summary
Bidirectional Projection Network for Cross Dimension Scene Understanding
Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, Tien-Tsin Wong
TL;DR
Existing systems often process 2D images and 3D point clouds separately despite their complementary texture and geometry information. BPNet connects symmetric 2D and 3D subnetworks with bidirectional projection modules at multiple levels, and achieves top ScanNetV2 performance for both segmentation tasks.
Problem
2D and 3D visual representations contain complementary information, but most recognition systems process them individually and existing joint methods largely use unidirectional fusion.
Method
BPNet uses symmetric 2D and 3D subnetworks connected by bidirectional projection modules that exchange multi-view features at multiple architectural levels.
Results
BPNet achieves top performance on ScanNetV2 for both 2D and 3D semantic segmentation and consistently outperforms single-domain baselines.
Takeaways & Limitations
Joint optimization over 2D and 3D scenes benefits both domains and produces more consistent 2D and 3D results.
Abstract
from arXiv · showhide
2D image representations are in regular grids and can be processed efficiently, whereas 3D point clouds are unordered and scattered in 3D space. The information inside these two visual domains is well complementary, e.g., 2D images have fine-grained texture while 3D point clouds contain plentiful geometry information. However, most current visual recognition systems process them individually. In this paper, we present a \emph{bidirectional projection network (BPNet)} for joint 2D and 3D reasoning in an end-to-end manner. It contains 2D and 3D sub-networks with symmetric architectures, that are connected by our proposed \emph{bidirectional projection module (BPM)}. Via the \emph{BPM}, complementary 2D and 3D information can interact with each other in multiple architectural levels, such that advantages in these two visual domains can be combined for better scene recognition. Extensive quantitative and qualitative experimental evaluations show that joint reasoning over 2D and 3D visual domains can benefit both 2D and 3D scene understanding simultaneously. Our \emph{BPNet} achieves top performance on the ScanNetV2 benchmark for both 2D and 3D semantic segmentation. Code is available at \url{https://github.com/wbhu/BPNet}.
1. Introduction
BPNet jointly reasons over complementary 2D and 3D scene representations by enabling bidirectional feature interaction across multiple network levels. Evaluations report benefits for both 2D and 3D semantic segmentation, including top performance on ScanNetV2.
- 2D images provide fine-grained texture and color, while 3D point clouds provide shape and geometry information.
- BPNet addresses limited 2D–3D fusion by enabling complementary information to flow bidirectionally at the network architectural level.
- BPNet uses similar U-Net structures for 2D and 3D data and fuses multi-view features end-to-end for joint scene understanding.
- The Bidirectional Projection Module transfers 2D and 3D features through projection links at multiple decoder pyramid levels.
- Joint modeling yields sharper 2D object boundaries and helps distinguish objects with similar shapes in 3D segmentation.
- BPNet achieves top ScanNetV2 performance for both 2D and 3D semantic segmentation and consistently outperforms single-domain baselines.
2. Related Work
Prior work develops separate 2D and 3D recognition methods and several approaches for combining their data. These fusion methods are predominantly unidirectional or rely on specialized representations and operations.
- 2D semantic segmentation: 2D segmentation methods use convolutional architectures, receptive-field expansion, contextual information, and attention, but lack geometric information.
- 3D semantic segmentation: 3D segmentation methods address irregular point clouds with point-based networks, local feature enhancement, kernel-based convolutions, and graph reasoning.
- Recognition with combined 2D-3D data: Existing combined 2D–3D methods commonly fuse information unidirectionally to improve recognition in one domain.
- Recognition with combined 2D-3D data: BPNet instead connects symmetric 2D UNet and 3D MinkowskiUNet subnetworks through bidirectional interaction at multiple decoder levels.
- Recognition with combined 2D-3D data: Other approaches encode images and camera parameters into 3D voxels or high-dimensional lattices and use specialized fusion operations.
3. Methodology
BPNet jointly processes 2D images and 3D scenes through symmetric sub-networks connected by bidirectional projection modules. The modules construct masked 2D–3D links and exchange features across decoder levels, while multi-view features are fused for simultaneous semantic predictions.
- Network architecture: BPNet uses 2D and 3D U-shaped sub-networks connected by BPM skip connections at corresponding decoder levels.The 2D branch uses conventional convolutions and the 3D branch uses sparse 3D convolutions.
- Link matrix construction: The camera matrix projects homogeneous 3D voxel coordinates to homogeneous 2D pixel coordinates before link construction.The camera matrix is the product of intrinsic calibration and extrinsic pose matrices.
- Link matrix construction: Depth comparison and view-frustum bounds determine whether each projected voxel–pixel correspondence is valid.A binary mask records validity, while occluded voxels are rejected using the projected depth and available depth map.
- Link matrix construction: BPM constructs an N × 3 link matrix containing projected 2D coordinates and a binary validity mask for each 3D voxel.The mask handles voxels without corresponding pixels, including those outside the view frustum or hidden by occlusion.
- Bidirectional projection: At multiple decoder levels, remapped links project 3D features into 2D and back-project 2D features into 3D before concatenation and 1 × 1 convolution.The link coordinates are remapped to match feature-map resolutions after down-sampling.
- View fusion: For multiple views, two sparse-convolution layers learn per-view impact factors and combine back-projected features by weighted summation.This replaces simple max-pooling for multi-view feature aggregation.
4. Experimental Evaluation
BPNet is evaluated on ScanNetV2 for joint 2D and 3D semantic segmentation, with comparisons against established baselines and ablations of projection design, view count, and voxel size. Results show benefits from bidirectional multi-level fusion, while performance also depends on the number of views and voxel resolution.
- ScanNetV2 benchmark: BPNet outperforms MinkowskiNet by 1.3 mIoU on ScanNetV2 3D semantic segmentation and other joint 2D-3D methods by at least 10.5 mIoU.It also exceeds point-based methods by at least 6.5 mIoU.
- ScanNetV2 benchmark: BPNet outperforms 2D-only methods by at least 18.1 mIoU on ScanNetV2 2D semantic segmentation while using the same model for its 2D and 3D results.The comparison indicates that 3D information can boost 2D segmentation, although the authors note the comparison is not completely fair because BPNet uses 3D information.
- Projection-level ablation: Bidirectional projection at all four pyramid levels outperforms single-level variants and the UNet34 and MinkowskiUNet18A baselines for both 2D and 3D segmentation.The ablation attributes the gain to integrating low- and high-level complementary information from both domains.
- Projection-direction ablation: Bidirectional projection performs best for both 2D and 3D results, exceeding unidirectional projection variants and the 2D-only or 3D-only baselines.The comparison supports mutual feature interaction between the two segmentation streams.
- View-count ablation: One 2D view performs best for 2D segmentation, whereas 3D segmentation improves from one to three views and declines slightly at five views.The authors attribute this trade-off to insufficient information with too few views and redundant information with too many.
- Voxel size and generalization: Decreasing voxel size improves 3D segmentation and also greatly improves 2D segmentation, indicating that higher-quality 3D information can boost the 2D CNN.BPNet is additionally evaluated on NYUv2 2.5D RGB-D data, where it performs favorably against typical RGB-D and joint 2D-3D baselines.
5. Conclusion
BPNet jointly performs 2D and 3D semantic segmentation by enabling bidirectional feature interaction across multiple pyramid levels. It achieves top ScanNetV2 performance and produces more consistent results across both domains.
- BPNet jointly performs 2D and 3D semantic segmentation using bidirectional feature interaction between 2D and 3D CNNs.The interaction occurs at multiple pyramid levels through the bidirectional projection module.
- BPNet leverages complementary advantages from 2D and 3D data to integrate their strengths for scene recognition.
- BPNet achieves top performance on the ScanNetV2 benchmark and consistently outperforms a baseline using a single 2D or 3D network.
- BPNet produces more consistent 2D and 3D segmentation results.