Source-linked AI summary
End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds
Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, Vijay Vasudevan
TL;DR
3D LiDAR detectors face a trade-off between BEV's shape-preserving but sparse representation and perspective view's dense but distance-dependent representation. The paper proposes point-level MVF with dynamic voxelization to combine both views, and reports significantly improved detection accuracy on Waymo and KITTI over comparable single-view PointPillars.
Problem
LiDAR detectors need to combine BEV's shape-preserving representation with perspective view's dense observations because each view has distinct limitations.
Method
MVF uses dynamic voxelization and point-level feature fusion to combine BEV and perspective context from the same LiDAR point cloud.
Results
MVF and dynamic voxelization significantly improve detection accuracy on the Waymo Open Dataset and KITTI dataset over the comparable single-view PointPillars baseline.
Takeaways & Limitations
Dynamic voxelization preserves the complete raw point cloud, yields deterministic voxel features, and provides a foundation for fusing information across views.
Takeaways & Limitations
Hard voxelization's fixed-capacity design illustrates the paper's scope boundary: point and voxel capacities can force information loss, unstable embeddings, and unnecessary computation.
Abstract
from arXiv · showhide
Recent work on 3D object detection advocates point cloud voxelization in birds-eye view, where objects preserve their physical dimensions and are naturally separable. When represented in this view, however, point clouds are sparse and have highly variable point density, which may cause detectors difficulties in detecting distant or small objects (pedestrians, traffic signs, etc.). On the other hand, perspective view provides dense observations, which could allow more favorable feature encoding for such cases. In this paper, we aim to synergize the birds-eye view and the perspective view and propose a novel end-to-end multi-view fusion (MVF) algorithm, which can effectively learn to utilize the complementary information from both. Specifically, we introduce dynamic voxelization, which has four merits compared to existing voxelization methods, i) removing the need of pre-allocating a tensor with fixed size; ii) overcoming the information loss due to stochastic point/voxel dropout; iii) yielding deterministic voxel embeddings and more stable detection outcomes; iv) establishing the bi-directional relationship between points and voxels, which potentially lays a natural foundation for cross-view feature fusion. By employing dynamic voxelization, the proposed feature fusion architecture enables each point to learn to fuse context information from different views. MVF operates on points and can be naturally extended to other approaches using LiDAR point clouds. We evaluate our MVF model extensively on the newly released Waymo Open Dataset and on the KITTI dataset and demonstrate that it significantly improves detection accuracy over the comparable single-view PointPillars baseline.
1 Introduction
MVF fuses BEV and perspective representations of the same LiDAR point cloud at the point level, while dynamic voxelization preserves points and supports stable, efficient multi-view encoding. The method improves detection accuracy on Waymo and KITTI.
- BEV preserves canonical object shapes but becomes sparse at long ranges, whereas perspective views are dense yet distort shape with distance.
- MVF fuses complementary BEV and perspective information from the same LiDAR sensor through point-level representations.Each point acts as the conduit for sharing context across views.
- Dynamic voxelization avoids predefined point sampling and padding, preserving points while reducing memory and computation overhead.It also avoids stochastic point or voxel dropout and produces deterministic voxel embeddings.
- Dynamic voxelization establishes point-level relationships across views, allowing each point to incorporate context from different local neighborhoods.
- MVF and dynamic voxelization significantly improve detection accuracy on the Waymo Open Dataset and KITTI dataset.
2 Related Work
Prior work mainly processes LiDAR point clouds through BEV projections, while other approaches combine multiple views or modalities. Figure 1 contrasts hard and dynamic voxelization as a concrete example of the latter distinction.
- Figure 1 depicts four voxels containing 6, 4, 2, and 1 points, comparing hard voxelization's dropped data with dynamic voxelization's full coverage.
- Hard voxelization can outperform fixed-buffer efficiency only by sampling or missing points and voxels when capacity is insufficient.The example uses 15F memory for hard voxelization versus 13F for dynamic voxelization.
- BEV-based LiDAR detectors transform point clouds into multi-channel 2D pseudoimages for 2D CNN processing and 3D detection.
- Multi-modal methods combine LiDAR views with camera information, using strategies such as ROI pooling, frustum processing, or feature interpolation.
3 Multi-View Fusion
MVF fuses BEV and perspective representations of the same LiDAR point cloud through point-level feature sharing. Dynamic voxelization preserves complete point-voxel mappings, avoids hard-voxelization losses, and supports context-aware fusion across views.
- Multi-View Fusion: MVF applies dynamic voxelization in BEV and perspective views, establishing bidirectional point-voxel mappings for cross-view feature fusion.Each point can reference corresponding voxel information from both views.
- Dynamic Voxelization: Hard voxelization samples points and voxels into fixed capacities and pads unused entries, causing information loss, unstable embeddings, and unnecessary computation.Its fixed buffer uses maximum voxel and point capacities, even when point density varies substantially.
- Dynamic Voxelization: Dynamic voxelization preserves complete point-voxel relationships with dynamic numbers of voxels and points, eliminating fixed buffers and stochastic dropout.Because raw point and voxel information is retained, voxel embeddings are deterministic and detection outcomes are more stable.
- Multi-View Representations: The perspective representation generalizes voxels to 3D frustums using spherical coordinates, exposing each point to a different local neighborhood than Cartesian BEV.These complementary neighborhoods provide distinct context for the same LiDAR point.
- Feature Fusion: The network embeds points with shared features, processes view-dependent voxel features through pooling and a convolution tower, then fuses three feature sources per point.The three sources are Cartesian voxel features, spherical voxel features, and shared point-wise features.
4 Experimental Results
Experiments on Waymo and KITTI compare hard-voxel, dynamic-voxel, and multi-view systems, showing that dynamic voxelization and MVF improve detection, particularly for distant and small objects.
- Experimental setup: The study compares HV+SV, DV+SV, and MVF using a common PointPillars backbone for fair evaluation.HV+SV and DV+SV learn 64D point embeddings, while MVF output is reduced to 64D.
- Datasets: The Waymo Open Dataset contains 1000 sequences spanning multiple cities, weather conditions, times of day, LiDARs, and cameras.Its training split has 798 sequences and its validation split has 202 sequences.
- Waymo evaluation: Waymo evaluates vehicle and pedestrian detection across overall, 0-30m, 30-50m, and >50m ranges using standard AP metrics.The evaluation uses 3D and BEV boxes with class-specific IoU thresholds.
- Waymo results: DV+SV consistently matches or improves HV+SV, while MVF further improves detection across ranges, especially for pedestrians and distant objects.MVF’s gains become more pronounced as detection distance increases, including more accurate long-range detections of occluded objects.
- Latency: For vehicle detection, MVF, DV+SV, and HV+SV run at 65.2ms, 41.1ms, and 41.1ms per frame, respectively.For pedestrian detection, their latencies are 60.6ms, 34.7ms, and 36.1ms per frame, respectively.
- KITTI results: On KITTI vehicle detection, dynamic voxelization improves accuracy over hard voxelization, and MVF further improves performance significantly.MVF also achieves competitive accuracy against other top-performing methods and can enhance other LiDAR detectors.
5 Conclusion
The paper introduces MVF, an end-to-end framework that fuses BEV and perspective views through dynamic voxelization for LiDAR 3D detection.
- Conclusion: MVF preserves raw points, produces deterministic voxel features, and uses point-level features to fuse complementary information across views.Experiments on Waymo and KITTI report significantly improved detection accuracy.