Source-linked AI summary
LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion
Xin Li, Tao Ma, Yuenan Hou, Botian Shi, Yuchen Yang, Youquan Liu, Xingjiao Wu, Qin Chen, Yikang Li, Yu Qiao, Liang He
TL;DR
Global LiDAR-camera fusion lacks fine-grained region-level information for 3D detection. LoGoNet combines global and local fusion with dynamic feature aggregation, and achieves state-of-the-art results on WOD and KITTI, including first place on Waymo with 81.02 mAPH (L2).
Problem
Global LiDAR-camera fusion across whole scenes lacks fine-grained local information, although images can complement sparse LiDAR point clouds with semantic and texture information.
Method
LoGoNet performs local-to-global fusion using GoF, LoF, and FDA to integrate multi-level voxel, image, and local point-position features.
Results
LoGoNet outperforms previous methods on WOD and KITTI and ranks first on the Waymo 3D detection leaderboard with 81.02 mAPH (L2).
Takeaways & Limitations
The framework provides local-to-global cross-modal feature fusion for richer information and accurate 3D detection across both benchmarks.
Takeaways & Limitations
The image branch is frozen, and only the LiDAR branch is trained.
Abstract
from arXiv · showhide
LiDAR-camera fusion methods have shown impressive performance in 3D object detection. Recent advanced multi-modal methods mainly perform global fusion, where image features and point cloud features are fused across the whole scene. Such practice lacks fine-grained region-level information, yielding suboptimal fusion performance. In this paper, we present the novel Local-to-Global fusion network (LoGoNet), which performs LiDAR-camera fusion at both local and global levels. Concretely, the Global Fusion (GoF) of LoGoNet is built upon previous literature, while we exclusively use point centroids to more precisely represent the position of voxel features, thus achieving better cross-modal alignment. As to the Local Fusion (LoF), we first divide each proposal into uniform grids and then project these grid centers to the images. The image features around the projected grid points are sampled to be fused with position-decorated point cloud features, maximally utilizing the rich contextual information around the proposals. The Feature Dynamic Aggregation (FDA) module is further proposed to achieve information interaction between these locally and globally fused features, thus producing more informative multi-modal features. Extensive experiments on both Waymo Open Dataset (WOD) and KITTI datasets show that LoGoNet outperforms all state-of-the-art 3D detection methods. Notably, LoGoNet ranks 1st on Waymo 3D object detection leaderboard and obtains 81.02 mAPH (L2) detection performance. It is noteworthy that, for the first time, the detection performance on three classes surpasses 80 APH (L2) simultaneously. Code will be available at \url{https://github.com/sankin97/LoGoNet}.
1. Introduction
LoGoNet addresses the limited fine-grained information of global LiDAR-camera fusion by combining global and proposal-level local fusion. Its fused features improve 3D detection, reaching state-of-the-art results on WOD and KITTI, including 81.02 mAPH (L2) on Waymo.
- Motivation: Global fusion across the whole scene lacks fine-grained local information because foreground objects occupy only a small part of the scene.The paper states that global fusion therefore brings only marginal gains.
- Approach: LoGoNet performs LiDAR-camera fusion at both global and local levels through GoF, LoF, and FDA modules.LoF complements GoF with region-level information, while FDA interacts between globally and locally fused features.
- Approach: GoF uses point centroids to represent voxel-feature positions more accurately for cross-modal alignment.It adaptively fuses image features through deformable cross-attention and uses ROI pooling to generate ROI-grid features.
- Approach: LoF divides proposals into uniform grids, projects grid centers onto images, and fuses sampled image features with position-encoded local point-cloud features.Cross-attention combines the sampled image features and encoded local grid features.
- Results: 81.02 mAPH (L2) places LoGoNet first on the Waymo 3D detection leaderboard, while the method achieves state-of-the-art performance on WOD and KITTI.The paper also reports that detection performance on three classes surpasses 80 APH (L2) simultaneously for the first time.
2. Related Work
Prior work spans image-based, LiDAR-based, and multi-modal 3D detection. Multi-modal methods evolved from proposal-level feature concatenation and detection-result combination toward global cross-modal enhancement of point-cloud features.
- Image-based 3D Detection: Image-based 3D detection methods estimate depth or lift 2D features into 3D because image depth is not directly accessible.The cited passage notes that image-based performance remains inferior to LiDAR-based approaches.
- LiDAR-based 3D Detection: LiDAR-based approaches are categorized as point-based, voxel-based, or point-voxel fusion methods according to their point-cloud representations.Voxel-based methods convert point clouds into voxels and use 3D sparse convolution layers to extract voxel features.
- Multi-modal 3D Detection: Early multi-modal methods independently extract two modalities and concatenate features at the proposal level, while CLOCs combines pre-trained 2D and 3D detection results.The passage describes these approaches as suffering from coarse feature aggregation or lacking integrated feature fusion.
- Multi-modal 3D Detection: Later methods globally enhance point-cloud features using semantic scores, image features, learned calibration, or fusion in a shared representation space.The passage identifies point decoration, 3D-CVF, EPNet, and subsequent global-fusion studies as examples.
3. Methodology
LoGoNet performs LiDAR-camera fusion globally and locally, then dynamically aggregates the resulting features for proposal refinement. Its global branch improves voxel-image alignment with point centroids, while its local branch preserves fine-grained proposal geometry and context.
- Framework overview: LoGoNet processes point clouds with a 3D voxel backbone and generates initial bounding-box proposals before multi-level cross-modal fusion.The image branch supplies multi-camera image features, and the final fusion combines voxel, image, and local position information during two-stage refinement.
- Global Fusion: The Global Fusion module represents non-empty voxel features with point centroids, projects them into images, and adaptively samples image features through learnable offsets.The sampled image features are fused with voxel features through cross-attention, followed by ROI pooling to produce ROI-grid features.
- Local Fusion: The Local Fusion module divides each proposal into u × u × u grids and encodes each grid’s relative position and point count with the Position Information Encoder.Grid centers are projected to multi-view images, where dynamically sampled image features are fused with encoded local grid features through cross-attention.
- Feature Dynamic Aggregation: The Feature Dynamic Aggregation module applies self-attention across grid points to build relationships between locally and globally fused features.The aggregated features are then flattened and used to refine bounding boxes.
4. Experiments
Experiments on WOD and KITTI evaluate LoGoNet under standard benchmark metrics, compare it with state-of-the-art methods, and ablate its fusion components and position information.
- Datasets and metrics: WOD uses AP and APH at LEVEL 1 and LEVEL 2, with mAPH (L2) as the Waymo leaderboard’s main ranking metric.KITTI reports AP at easy, moderate, and hard levels.
- Waymo results: 81.02 mAPH (L2) places LoGoNet Ens first on the Waymo 3D detection leaderboard and above 80 APH (L2) for vehicle, pedestrian, and cyclist simultaneously.The ensemble result surpasses BEVFusion TTA by 1.05 mAPH (L2).
- Waymo results: 77.10 mAPH (L2) makes nonensembled LoGoNet the strongest submitted nonensembled method, exceeding BEVFusion by 0.77% and MPPNet with 16 frames by 1.43%.LoGoNet with three frames surpasses MPPNet with 16 frames, while five frames surpasses it by 0.69% mAPH (L2).
- KITTI results: LoGoNet achieves state-of-the-art KITTI results, surpassing PDV by 1.47% mAP on validation and SFD by 1.07% mAP on test car detection.On the KITTI test set, it achieves state-of-the-art mAP at three difficulty levels for car and cyclist.
- Fusion gains: The proposed fusion improves WOD mAPH (L2) by +0.81%, +2.93%, and +2.25% for vehicle, pedestrian, and cyclist, respectively.On KITTI, gains are +0.70%, +4.83%, and +3.66% mAP for car, pedestrian, and cyclist.
- Ablation studies: GoF, LoF, and FDA together exceed the single-modal RCNN-only module by 1.85%, 3.19%, and 1.63% APH (L2) for vehicle, pedestrian, and cyclist.FDA alone contributes 0.55%, 0.25%, and 0.46% APH (L2), respectively.
- Ablation studies: Richer LoF grid position information improves APH (L2) by 0.12%, 0.28%, and 0.15% for vehicle, pedestrian, and cyclist, respectively.Table 6 evaluates combinations of spatial locations, point counts, and point centroids.
5. Conclusion
LoGoNet integrates point-cloud and image features through local-to-global cross-modal fusion for 3D detection, outperforming previous methods on WOD and KITTI.
- LoGoNet deeply integrates point-cloud and image features through local-to-global cross-modal fusion to provide richer information for accurate detection.
- Extensive experiments on WOD and KITTI show that LoGoNet surpasses previous methods on both benchmarks and ranks first on the Waymo 3D detection leaderboard.