Source-linked AI summary
M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers
Tianrui Guan, Jun Wang, Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, Dinesh Manocha
TL;DR
3D object detection needs to combine heterogeneous point-cloud representations, feature scales, and mutual point relationships. M3DETR uses stacked transformers to unify these factors, achieving state-of-the-art results on KITTI and Waymo, including a 1.48% mAP improvement for all classes on Waymo.
Problem
Existing methods face unresolved challenges when fusing voxel, raw-point, and bird’s-eye-view representations and when combining multi-scale features with conflicting resolutions and receptive fields.
Method
M3DETR uses stacked M3 Transformers to jointly model multi-representation, multi-scale, and mutual-relation features for 3D object detection.
Results
1.48% mAP improvement for all classes on the Waymo test set accompanies state-of-the-art performance on the KITTI and Waymo datasets.
Takeaways & Limitations
The unified architecture combines representation, scale, and relationship modeling while remaining invariant to transformer hyper-parameters.
Abstract
from arXiv · showhide
We present a novel architecture for 3D object detection, M3DeTR, which combines different point cloud representations (raw, voxels, bird-eye view) with different feature scales based on multi-scale feature pyramids. M3DeTR is the first approach that unifies multiple point cloud representations, feature scales, as well as models mutual relationships between point clouds simultaneously using transformers. We perform extensive ablation experiments that highlight the benefits of fusing representation and scale, and modeling the relationships. Our method achieves state-of-the-art performance on the KITTI 3D object detection dataset and Waymo Open Dataset. Results show that M3DeTR improves the baseline significantly by 1.48% mAP for all classes on Waymo Open Dataset. In particular, our approach ranks 1st on the well-known KITTI 3D Detection Benchmark for both car and cyclist classes, and ranks 1st on Waymo Open Dataset with single frame point cloud input. Our code is available at: https://github.com/rayguan97/M3DETR.
1. Introduction
M3DETR addresses the difficulty of fusing point-cloud representations, feature scales, and point relationships by using transformers in a unified 3D detection architecture. It reports improved performance and state-of-the-art results on major benchmarks.
- 3D object detection supports applications including autonomous driving, augmented reality, and robotics.
- Existing methods use voxel, raw-point, or bird’s-eye-view representations, but fusing them is non-trivial because their architectures and inputs create semantic gaps.
- Multi-scale feature fusion is difficult because higher resolution and larger receptive fields conflict, while correlations between feature maps at different scales remain challenging.
- Transformers are motivated as a way to narrow semantic gaps by adapting to informative features and eliminating noise.
- M3DETR is a unified transformer architecture that models multi-representation, multi-scale, and mutual-relation features end to end.
- 1.48% mAP improvement for all classes on the Waymo test set is reported alongside state-of-the-art performance on KITTI and Waymo.
2. Related Work
Prior work develops separate strategies for BEV, voxel, point-wise, multi-scale, and mutual-relation modeling. M3DETR combines these representations and scales with transformers to address representation gaps and capture richer point-cloud relationships.
- Multi-representation modeling: BEV methods project point clouds into 2D space, while voxel-based architectures encode irregular point clouds using equally spaced 3D voxels.
- Multi-scale modeling: Multi-scale modeling enlarges receptive fields and increases resolution, with prior 3D methods using multiple sampling ranges or detection heads.
- Mutual-relation modeling: Point-cloud methods model local context and mutual relations using grouping, convolution-like operators, and graph-based approaches.
- M3DETR architecture: M3DETR uses M3 Transformers for inter- and intra-feature attention across representations, scales, and locations.
- Transformer-based modeling: Unlike prior point-based transformers focused mainly on local or global point context, M3DETR also models multi-scale and multi-representation features.
- M3DETR contribution: The proposed model addresses voxel-wise, point-wise, and BEV-wise feature gaps while capturing deeper local-global structures and geometric relationships.
3. Our Approach
M3DETR encodes raw point clouds into voxel, point, and BEV representations, fuses them across representations and scales with transformers, models point-wise relations, and performs coarse-to-fine 3D detection.
- Feature embeddings: M3DETR generates voxel-, point-, and BEV-based feature embeddings from raw point clouds before transformer fusion.Voxel features use voxelization and 3D sparse convolutions at multiple scales.
- M3 Transformers: The M3 transformer dynamically attends across voxel, point, and BEV embedding sequences to produce cross-representation, cross-scale, and cross-point features.The architecture uses stacked multi-representation/multi-scale and mutual-relation transformer components.
- M3 Transformers: The multi-representation and multi-scale transformer aggregates six aligned inputs spanning point, BEV, and voxel features at 1×, 2×, 4×, and 8× scales.Feature reduction first aligns differing embedding dimensions, then self-attention produces corresponding representations.
- M3 Transformers: The mutual-relation transformer applies multi-head self-attention to point-wise keypoint features to model relationships among keypoints.Aggregated features from the first transformer block are concatenated and split into point-wise features before this stage.
- Detection heads: The detection heads use RPN and R-CNN stages to predict 3D box class, localization, and orientation in a coarse-to-fine manner.RPN generates proposals from BEV features, while R-CNN refines them using proposal features and 3D IoU-guided confidence scoring.
- Loss functions: Training combines classification, box-regression, refinement, and 3D IoU losses for both detection stages.The classification term uses focal loss, while localization regression uses Smooth L1 loss and IoU supervision uses 3D IoU loss.
4. Experiments
M3DETR is evaluated qualitatively and quantitatively on Waymo and KITTI LiDAR-based 3D detection benchmarks, including robustness and ablation studies. It achieves strong benchmark results while improving over PV-RCNN and benefiting from jointly modeling representations, scales, and relations.
- Evaluation setup: M3DETR is evaluated on Waymo Open Dataset and KITTI for LiDAR-based 3D object detection, with experiments covering accuracy and hyperparameter robustness.The evaluation includes vehicle, pedestrian, cyclist, car, and cyclist classes under dataset-specific difficulty protocols.
- Waymo results: 1.48% and 1.85% improvements over PV-RCNN are reported for average LEVEL 1 and LEVEL 2 mAP, respectively, on Waymo Vehicle detection.The comparison uses a reproduced PV-RCNN baseline with single-frame input.
- Waymo results: M3DETR achieves state-of-the-art Vehicle performance at both LEVEL 1 and LEVEL 2 among published single-frame LiDAR methods.The results are compared with PV-RCNN in qualitative visualizations and benchmark evaluations.
- KITTI results: 1.24% improvement is reported over HotSpotNet’s 82.59% easy-category result for KITTI 3D cyclist detection.The paper reports a large-margin advantage especially for the cyclist class.
- Ablation studies: 4.07% and 1.71% gains over PV-RCNN are obtained with a single multi-representation and multi-scale transformer layer at 11 and 40 recall positions.A single mutual-relation transformer layer yields 4.47% and 1.99% gains, while the unified model achieves the best performance without hyperparameter tuning.
- Robustness: M3DETR performs consistently well across sampling sizes, detection-head counts, and transformer-layer counts for KITTI Car detection.The robustness tests use IoU 0.7 and evaluate both 11 and 40 recall positions.
5. Conclusions
M3DETR is a transformer-based LiDAR detection framework that jointly models multiple representations, feature scales, and mutual relations. Its proposed transformers combine cross-representation and multi-scale information with point-wise interactions, yielding strong results on Waymo and KITTI while remaining robust to transformer hyperparameters.
- Framework: M3DETR is a transformer-based framework for LiDAR point-cloud object detection.The framework is designed to jointly model multi-representation, multi-scale, and mutual-relation features.
- Qualitative comparison: The visual comparison shows M3DETR results against PV-RCNN on the same Waymo input, highlighting PV-RCNN false negatives.The comparison is presented as a qualitative detection-result visualization.
- Architecture: The first M3 Transformer integrates features across scales and representations, while the second aggregates information from keypoints.The two transformers implement the framework’s representation-scale fusion and point-wise aggregation.
- Results: M3DETR outperforms previous work by a large margin on both the Waymo Open Dataset and KITTI dataset.The conclusion also reports invariance to transformer hyperparameters without additional bells and whistles.
A. More Our Approach Details
The section introduces further discussion of the approach but provides no substantive methodological detail in the supplied passage.
- Further discussion: The section signals that additional discussion of the approach follows.No specific component, operation, or result is described in the supplied text.
A.1. Voxel Representation in Multi-Representation Feature Embeddings
Voxel-wise feature extraction converts raw point clouds into discretized voxels and derives voxel features using sparse 3D convolutions.
- Voxelization: Voxelization converts raw point clouds into equally sized 3D voxels before feature extraction.Each discretized voxel is represented with dimensions L×W×H×C, where C is the voxel-feature channel dimension.
- Voxel features: The feature for each nonempty voxel is computed as the average of point-wise features from its points.This produces a voxel-level representation for subsequent processing.
- Feature extraction: Sparse 3D convolutions extract features after voxelization.The passage identifies voxelization and sparse-convolution feature extraction as the two processing steps.
A.2. Multi-head Self-attention Basics
Multi-head self-attention builds on attention by using N heads over input matrix X. Q, K, and V are learned linear projections of X, with learnable parameters associated with each head.
- MHSA is defined for an input matrix X using N attention heads.
- Q, K, and V are linear projections of the input matrix X.
- Each attention head uses learnable projection parameters with hidden dimensions for Q, K, and V.
B. More Implementation Details
The implementation section provides additional details intended to support reproduction. It states that code and trained models will be released to match the reported performance.
- The paper gives additional implementation details for reproducing its results.
- The authors state that code will be released later.
- The planned release includes trained models that can match the performance reported in the paper.
B.1. Backbone
The backbone combines a four-block 3D sparse-convolution voxel branch with point-cloud features to refine sampled keypoint features. Waymo and KITTI use different voxel sizes, spatial ranges, and 2D convolution output sizes.
- Backbone: The voxel CNN branch has four 3D sparse-convolution blocks with channel dimensions 16, 32, 64, and 64.
- Backbone: 2,048 keypoints are sampled for both Waymo and KITTI, using voxel and point features to refine keypoint features through PointNet.
- Waymo: Waymo uses voxel size [0.1, 0.1, 0.15] and a 188 × 188 × 512 2D-convolution output.
- KITTI: KITTI uses voxel size [0.05, 0.05, 0.1] and a 200 × 176 × 512 2D-convolution output.
B.2. Detection Heads
The detection heads initialize proposals with category-specific average anchor sizes and refine them using grid-based region pooling. Training and validation use different proposal counts and NMS thresholds.
- Detection Heads: RPN anchor sizes are set by averaging the corresponding annotated objects for each category.
- Detection Heads: RoI-grid pooling samples 6 × 6 × 6 grid points within each initial 3D proposal.
- Detection Heads: The grid-point features use M = 16 surrounding points.
- Detection Heads: Training sends 512 RPN proposals to R-CNN with NMS threshold 0.8, while validation sends 100 with threshold 0.7.