Source-linked AI summary
AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, Hang Zhao
TL;DR
3D detection needs to combine LiDAR geometry with RGB semantics, but aligning their non-homogeneous features remains challenging. AutoAlign learns this alignment through pixel-level cross-attention and instance-level interaction, with joint 2D–3D training. It improves multi-modal detection across KITTI and nuScenes, including a 7.0 mAP gain on nuScenes validation over CenterPoint.
Problem
Combining LiDAR geometry and RGB semantics remains challenging because the two modalities provide complementary but non-homogeneous information.
Method
AutoAlign uses learnable pixel-level cross-attention alignment, instance-level cross-modal interaction, and joint 2D–3D training for multi-modal feature fusion.
Results
7.0 mAP: AutoAlign outperforms the CenterPoint baseline by 7.0 mAP on the nuScenes validation set, reaching 66.6 mAP.
Takeaways & Limitations
AutoAlign provides a learnable multi-modal fusion approach that improves various 3D detectors on KITTI and nuScenes.
Takeaways & Limitations
CAFA captures fine-grained image features but fails to capture instance-level information without complementary instance-level interaction.
Abstract
from arXiv · showhide
Object detection through either RGB images or the LiDAR point clouds has been extensively explored in autonomous driving. However, it remains challenging to make these two data sources complementary and beneficial to each other. In this paper, we propose \textit{AutoAlign}, an automatic feature fusion strategy for 3D object detection. Instead of establishing deterministic correspondence with camera projection matrix, we model the mapping relationship between the image and point clouds with a learnable alignment map. This map enables our model to automate the alignment of non-homogenous features in a dynamic and data-driven manner. Specifically, a cross-attention feature alignment module is devised to adaptively aggregate \textit{pixel-level} image features for each voxel. To enhance the semantic consistency during feature alignment, we also design a self-supervised cross-modal feature interaction module, through which the model can learn feature aggregation with \textit{instance-level} feature guidance. Extensive experimental results show that our approach can lead to 2.3 mAP and 7.0 mAP improvements on the KITTI and nuScenes datasets, respectively. Notably, our best model reaches 70.9 NDS on the nuScenes testing leaderboard, achieving competitive performance among various state-of-the-arts.
1 Introduction
AutoAlign is an integrated multi-modal fusion framework that adaptively combines image and point-cloud features at pixel and instance levels. It adds joint 2D–3D training to regularize image features and reports competitive results on KITTI and nuScenes.
- LiDAR provides precise 3D geometry but lacks semantic information and suffers from sparse returns, while RGB images provide semantic and long-distance information.
- AutoAlign adaptively fuses non-homogeneous image and point-cloud representations through pixel-level aggregation and instance-level feature interaction.
- Each voxel queries the image plane to learn a pixel-wise alignment map, aggregates image features, and concatenates them with the original 3D features.
- Joint 2D–3D detection training regularizes image-branch features and improves detection accuracy.
- AutoAlign is evaluated across KITTI and nuScenes and achieves competitive performance on both datasets.
2 Related Work
Prior 3D detection methods process camera images, LiDAR data, or fused multi-modal representations. Multi-modal fusion includes decision-level, region-wise, projection-based, and point-wise strategies, each combining modalities at different stages.
- Camera-based methods estimate depth from images, whereas stereo methods generate dense point clouds for 3D detection.
- LiDAR detectors use voxel-, point-, or view-based representations, with point-level processing preserving geometry but generally requiring more computation.
- Multi-modal methods fuse detections or features through 2D-to-3D refinement, RoI-wise fusion, learned projection, or point-wise feature integration.
3 Method
AutoAlign aligns image and point features through learnable cross-modal fusion at pixel and instance levels. CAFA aggregates image features for voxel representations, while SCFI provides instance-level semantic guidance and joint training regularizes the image branch.
- 3 Method: AutoAlign replaces deterministic camera-projection alignment with adaptive cross-modal feature alignment for non-homogeneous image and voxel representations.Each voxel feature queries image features through a learnable alignment process.
- 3.1 Pixel-level Feature Aggregation: CAFA uses voxel features as queries and image features as keys and values to aggregate pixel-level information across the image plane.The resulting weighted aggregation enriches voxel features with global-view RGB information.
- 3.1 Pixel-level Feature Aggregation: The normalized attention weights form an alignment map that determines how image pixels contribute to each voxel feature.A feed-forward network produces the final RGB-aware point features after weighted aggregation.
- 3.2 Instance-level Feature Interaction: CAFA captures fine-grained image information but lacks instance-level information, whereas RoI-wise fusion preserves object integrity while providing coarser aggregation without 2D information during proposal generation.SCFI is introduced to bridge these complementary limitations.
- 3.2 Instance-level Feature Interaction: SCFI forms paired 2D and 3D RoI features and minimizes their cross-modal feature distance with self-supervised projection and prediction heads.The method applies a symmetric interaction loss to pull paired modal representations closer.
- 3.3 Joint Training for 2D-3D Detection: The joint-training framework combines 3D detection, 2D detection, and SCFI objectives to regularize image representations during 2D-3D detection.The image branch receives explicit 2D detection supervision rather than relying only on implicit supervision from the 3D branch.
4.1 Implementation Details
Experiments instantiate AutoAlign across representative 3D detectors and specify the image branch, alignment modules, and joint-training configuration.
- AutoAlign is evaluated with PointPillar, SECOND, and CenterPoint as representative 3D detectors.
- Faster R-CNN with ResNet50 serves as the image-branch detector.
- The cross-attention alignment module uses 128 hidden units, while 2DRoIAlign and 3DRoIPooling each output size 4.
- The self-supervised cross-modal module uses 2048-unit projector and predictor MLPs with 512 hidden units.
4.2 Results on KITTI dataset
On KITTI, AutoAlign improves representative pillar- and voxel-based detectors, with the largest gains occurring for objects whose LiDAR reflections are sparse.
- 3.0 and 2.3 mAP improvements are obtained for PointPillar and SECOND, respectively, under the 3D moderate evaluation protocol.
- 3.0 and 3.5 mAP gains are achieved for Pedestrian and Cyclist on AP3D moderate, respectively.
- AutoAlign benefits objects with sparse reflections by adding RGB information that is dense and rich in semantic and texture information.
4.3 Results on NuScenes dataset
On nuScenes, AutoAlign substantially outperforms CenterPoint and surpasses MVP under matched single-stage settings, while retaining a simple joint-training design.
- 66.6 mAP and 71.1 NDS are achieved on the nuScenes validation set, exceeding CenterPoint by 7.0 mAP and 6.5 NDS.
- AutoAlign surpasses MVP by 1.1 NDS under the same single-stage settings.
- The method requires neither sophisticated virtual point generation nor image feature pre-fetching, supporting suitability for real-world applications.
4.4 Ablation Studies
Ablations show complementary gains from pixel-level alignment, instance-level feature interaction, and 2D joint training on KITTI with SECOND.
- The ablation results are reported on the KITTI validation set using SECOND as the baseline detector.
- Cross-attention feature alignment raises accuracy by 0.5 mAP across all difficulty levels.The result supports preserving high-resolution image information during cross-modal aggregation.
- Adding SCFI brings a 1.2 mAP enhancement, increasing overall moderate AP3D from 68.5 to 69.7.SCFI provides instance-level supervision for aligning semantically paired features.
- Adding 2D joint training provides another 0.6 mAP gain and raises APhard by 1.0 mAP.The gains are attributed to image-backbone regularization and reduced training mismatch between 2D and 3D models.
4.5 Discussions
The discussions compare cross-modal query strategies and feature sources for AutoAlign, then examine self-supervised losses and alignment-map quality. These analyses motivate single-head cross-attention, C5 image features, and negative cosine similarity with positive pairs.
- Investigating the Best Cross-modal Query Strategy.: Single-head cross-attention is selected as the cross-modal query strategy because it balances computational cost and efficiency.The alternatives include projection with bilinear interpolation, a non-local block, and self-attention-like cross-modal querying.
- Visualization and Analysis: SCFI supplies instance-level semantic supervision that makes CAFA alignment maps positionally and semantically meaningful.Without feature interaction, CAFA fails to produce meaningful alignment maps; with SCFI, it produces reasonable maps on 2D images.
- Seeking the Suitable Feature Source for Self-supervised Feature Interaction.: C5 image features outperform P5 features for cross-modal feature interaction.C5 comes directly from the ResNet backbone, whereas P5 is produced after the feature pyramid network.
- Seeking the Suitable Feature Source for Self-supervised Feature Interaction.: Negative cosine similarity with positive pairs is chosen for self-supervised cross-modal learning.The authors avoid negative pairs because similar instances may have comparable shapes, while point features contain less identity information than images.
4.6 Visualization and Analysis
The visualization compares CAFA alignment maps with and without SCFI for randomly selected point voxels. SCFI enables alignment maps that are positionally and semantically reasonable, whereas CAFA alone fails to do so.
- Visualization and Analysis: Without SCFI, CAFA fails to yield a meaningful alignment map on the 2D image.The comparison uses alignment maps generated from different 3D query voxel features.
- Visualization and Analysis: With SCFI, CAFA produces a positionally and semantically reasonable feature alignment map.SCFI provides instance-level semantic supervision during feature alignment.
5 Conclusion
The conclusion presents AutoAlign as a learnable multi-modal fusion method for 3D object detection. CAFA aggregates image information per voxel, while SCFI enhances semantic consistency during feature assignment, and experiments show improvements on KITTI and nuScenes.
- Conclusion: AutoAlign is a learnable multi-modal feature fusion method for 3D object detection.The method is intended to provide a new perspective on multi-modal feature fusion for autonomous driving.
- Conclusion: CAFA enables each voxel feature to aggregate image information in a fine-grained manner.The module performs cross-attention feature alignment between voxel and image features.
- Conclusion: SCFI enhances semantic consistency during assignment for the CAFA module.It is designed as a self-supervised cross-modal feature interaction module.
- Conclusion: AutoAlign significantly improves various 3D detectors on the KITTI and nuScenes datasets.The conclusion attributes this result to comprehensive experimental evaluation.