Source-linked AI summary
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation
Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, Shiliang Pu
TL;DR
LiDAR point-cloud segmentation must reconcile complementary views with distinct geometric, sparsity, distortion, and quantization trade-offs. RPVNet fuses range, point, and voxel representations through adaptive interactions, achieving state-of-the-art results on SemanticKITTI and nuScenes while improving efficiency.
Problem
Point-based, voxel-based, and range-based representations each have limitations, including disordered neighborhoods, cubic high-resolution computation, physical distortion, and quantization loss in fine-grained segmentation.
Method
RPVNet uses a deep range-point-voxel fusion framework with mutual interactions, hash-mapped efficient feature propagation, and a gated fusion module that adaptively weights concurrent view features.
Results
RPVNet achieves state-of-the-art performance on both SemanticKITTI and nuScenes, with about 5% ∼12% performance gain than range-based methods.
Takeaways & Limitations
The proposed multi-view interaction paradigm supports adaptive cross-view enhancement and efficient LiDAR segmentation using relatively lower voxel resolution.
Takeaways & Limitations
Voxel-based representation remains relatively sparse and requires high resolution to reduce quantization loss, causing cubic increases in computation and memory footprint.
Abstract
from arXiv · showhide
Point clouds can be represented in many forms (views), typically, point-based sets, voxel-based cells or range-based images(i.e., panoramic view). The point-based view is geometrically accurate, but it is disordered, which makes it difficult to find local neighbors efficiently. The voxel-based view is regular, but sparse, and computation grows cubically when voxel resolution increases. The range-based view is regular and generally dense, however spherical projection makes physical dimensions distorted. Both voxel- and range-based views suffer from quantization loss, especially for voxels when facing large-scale scenes. In order to utilize different view's advantages and alleviate their own shortcomings in fine-grained segmentation task, we propose a novel range-point-voxel fusion network, namely RPVNet. In this network, we devise a deep fusion framework with multiple and mutual information interactions among these three views and propose a gated fusion module (termed as GFM), which can adaptively merge the three features based on concurrent inputs. Moreover, the proposed RPV interaction mechanism is highly efficient, and we summarize it into a more general formulation. By leveraging this efficient interaction and relatively lower voxel resolution, our method is also proved to be more efficient. Finally, we evaluated the proposed model on two large-scale datasets, i.e., SemanticKITTI and nuScenes, and it shows state-of-the-art performance on both of them. Note that, our method currently ranks 1st on SemanticKITTI leaderboard without any extra tricks.
1. Introduction
RPVNet addresses the complementary strengths and shortcomings of point-, voxel-, and range-based LiDAR representations through deep, adaptive multi-view fusion. Its efficient interactions and experiments support state-of-the-art results on SemanticKITTI and nuScenes.
- Representation trade-offs: Voxel views retain physical dimensions but require high resolution to reduce quantization loss, causing cubic growth in computation and memory.
- Representation trade-offs: Point views preserve geometric detail but make neighbor search inefficient because points are unstructured.
- Representation trade-offs: Range views use spherical projection, which distorts physical dimensions and can cause severe object overlap in cluttered scenes.
- Motivation: Voxel methods generally outperform point- and range-based methods, while range methods are more efficient and point methods can miss real-time requirements during neighbor search.
- Proposed approach: RPVNet uses points as an intermediate carrier, transfers range and voxel features to points, adaptively selects features, and propagates fused features back.
- Proposed approach: The framework performs multi-view fusion repeatedly rather than only at the network front or end, enabling deeper interactions among views.
- Efficiency and evaluation: The proposed hash-mapping-based RPV interaction mechanism is efficient and summarized in a general formulation for future extension.
- Efficiency and evaluation: RPVNet achieves state-of-the-art results on both SemanticKITTI and nuScenes.
2. Related Work
Related work develops point-, voxel-, range-, and multi-view approaches for LiDAR segmentation, but each representation retains important limitations. RPVNet is positioned as a multi-view alternative to these earlier fusion strategies.
- Point-based methods: Point-based methods process points directly, but unstructured data makes local-neighbor search inefficient and limits scalability for large outdoor scenes.
- Voxel-based methods: Voxel-based methods apply 3D convolutions to partitioned point clouds, with later work targeting lower computation and improved performance.
- Range-based methods: Range-based methods project point clouds into dense spherical grids for 2D CNN processing, but spherical projection distorts physical dimensions.
- Multi-view fusion: Multi-view methods combine representations through early fusion, late fusion, or point-voxel interaction, but their fusion designs differ in timing and mechanism.
3. Methodology
RPVNet combines voxel, point, and range branches through repeated interactions, using points as intermediate hosts for feature transfer and adaptive fusion. Its indexing and propagation mechanisms connect representations efficiently, while gated fusion filters view-specific information and instance mix addresses class imbalance.
- Framework Overview: RPVNet uses voxel, point, and range branches with multiple interactions; voxel and range branches are U-Net-like, while the point branch uses per-point MLPs.RPV fusion is applied after the stem, fourth down-sampling, second up-sampling, and last up-sampling stages.
- Efficient Multi-View Interactive Learning: Multi-view indexing maps points to voxel or range representations through projection and hashing, providing an efficient search connection between representations.The projection may be many-to-one, while the hash function is one-to-one in this formulation.
- Efficient Multi-View Interactive Learning: Feature propagation averages point features into other views and uses nearest-neighbor, trilinear, or bilinear interpolation to transfer features back to points.The propagation functions support point-to-view and inverse view-to-point information flow.
- Efficient Multi-View Interactive Learning: Points serve as intermediate hosts that receive voxel-cell and range-pixel features before fused features are propagated back to the other views.This design enables mutual feature exchange among the three representations within the network.
- Gated Fusion Module: Gated fusion adaptively weights concurrent view features because their importance varies and addition or concatenation can combine useful features with non-informative ones.Each gate is estimated with a convolutional layer and converted into probability weights through softmax after channel-wise votes are accumulated.
- Instance CutMix: Instance mix is introduced to address the imbalanced-class problem in LiDAR semantic segmentation, while the three views contribute complementary geometric, spatial, and semantic information.Point features preserve fine-grained geometry, voxels maintain spatial structure, and range images provide dense representations with large receptive fields.
4. Experiments
Experiments evaluate RPVNet on SemanticKITTI and nuScenes, comparing segmentation performance, efficiency, view combinations, fusion styles, ensembles, and component effects. The results report strong multi-view performance with acceptable overhead and gains from gated, three-view fusion.
- Benchmark evaluation: RPVNet is evaluated against state-of-the-art methods on SemanticKITTI and nuScenes using class-wise and mean IoU results.SemanticKITTI results use 0.05m voxels with instance CutMix; nuScenes reports RPVNet without instance CutMix.
- Efficiency: RPVNet maintains competitive running latency through compact network design and efficient implementation despite using multi-view features.Table 3 reports the mIoU–latency trade-off on SemanticKITTI, including preprocessing and post-processing time.
- Benchmark evaluation: RPVNet achieves state-of-the-art results on both SemanticKITTI and nuScenes, with about 5% ∼12% performance gain over range-based methods.The reported nuScenes comparison includes RangeNet++, PolarNet, SalsaNext, AMVNet, and Cylinder3D.
- Ablation studies: Across view ablations, RP, PV, and RPV fusion improve over their corresponding single-view baselines, while RPV fusion adds performance with acceptable complexity overhead.The experiments use one-quarter of the training data and test different voxel resolutions on SemanticKITTI and nuScenes.
- Ablation studies: RPV fusion can retain comparable performance at low voxel resolution, supporting its value for real-time applications.Table 4 examines voxel resolutions of 0.05m, 0.1m, and 0.3m.
- Ablation studies: Gated fusion improves mIoU over additive fusion by 1% on SemanticKITTI and 0.7% on nuScenes.The comparison is conducted without instance CutMix.
5. Conclusion
RPVNet is a deep and efficient fusion network for LiDAR segmentation, enabling three views to enhance one another adaptively. Extensive experiments report effectiveness and efficiency, including state-of-the-art results on two large-scale public datasets.
- RPVNet adaptively enhances LiDAR point-cloud representations across range, point, and voxel views.
- The method summarizes efficient multi-view interaction through representation indexing and feature propagation for point-level context exchange.
- RPVNet achieves state-of-the-art results on two large-scale public datasets.